About Openflow - Snowflake Deployments¶
Openflow - Snowflake Deployment run on Snowpark Container Services (SPCS) and provide a streamlined and integrated solution for data integration and connectivity across interoperable storage like Iceberg and Snowflake native storage. As a fully self-contained service within Snowflake, it’s easy to deploy and manage, offering a convenient and cost-effective environment for running your data flows. A key advantage is its native integration with Snowflake’s security model, which allows seamless authentication, authorization, and network security, and simplified operations.
Although customers can have both BYOC and Snowflake Deployments, the following list use cases that are well-suited to Snowflake Deployments:
Incorporating full-fidelity data in the bronze layer: Landing raw data from various sources directly into Snowflake and using Openflow Snowflake Deployments to extract and load.
Enriching data: Running pipelines to enrich tables that already exist inside Snowflake.
From ingest to insight in one place: Building applications where the entire data lifecycle (ingest, process and serve) happens within the Snowflake ecosystem.
Transforming raw data to insights with AI: Ingesting unstructured data and then, for instance, using Snowflake Intelligence to search and understand it better, all in concert with users’ other structured data.
Employing reverse ETL: Closing the loop on insight generation by sharing with external operational systems via APIs, messaging infrastructure, and more.
Understanding Runtime roles and External Access Integrations¶
Openflow - Snowflake Deployments must be able to interact with data sources and destinations that are typically outside Snowflake. In addition these deployments must also be able to communicate with and access Snowflake itself. Runtime roles and external access integrations provide this support.
What is a runtime role?¶
A runtime role is a traditional Snowflake role, associated with a specific Openflow Runtime, and used for the following tasks:
Grant access to external access integrations (EAIs). These EAIs specify rules that allow the runtime to access the data sources and destinations from within Snowflake itself.
Grant access to Snowflake resources.
Grant access to resources that are connector-specific
Runtime roles are linked to Openflow session tokens, avoiding the need for customers to create separate service users and key pairs for authentication to Snowflake.
What is an External Access Integration(EAI) within Openflow?¶
An External access integration (EAI) is a Snowflake object designed to provide secure access to external resources, like source systems from which Openflow connectors pull external data. Openflow Snowflake Deployments use EAIs and network rules together to define the endpoints an Openflow connector can read from or write to.
Data engineers define and configure EAIs and Runtime roles specific to a given connector and its underlying runtime.
Typical Openflow - Snowflake Deployment workflow¶
The following sections describe Openflow - Snowflake Deployment concepts and workflows.
User persona |
Task |
|---|---|
Snowflake administrator |
|
Data engineer (pipeline author, responsible for data ingestion) |
Connectors are a simple way to solve for a specific integration use case, and less technical users can deploy them without assistance from a data engineer. |
Data engineer (pipeline operator) |
Configures flow parameters and runs the flow. |
Data engineer (responsible for transformation to silver and gold layers) |
Responsible for transforming data from the bronze layer that was populated by the pipeline to silver and gold layers for analytics. |
Business user |
Makes use of gold layer objects for analytics. |
Limitations¶
As described in the Snowflake Openflow BYOC terms, securing Openflow BYOC is a shared responsibility model.
Openflow authorization uses roles and their associated privileges that are directly granted to the user. Currently, Openflow does not support authorization when the role is attached to another role within the user’s role hierarchy.