Set up the Openflow Connector for Microsoft Dataverse

Note

The connector is subject to the Connector Terms.

This topic describes the steps to set up the Openflow Connector for Microsoft Dataverse.

Prerequisites

  1. Ensure that you have reviewed Openflow Connector for Microsoft Dataverse.

  2. Ensure that you have set up Openflow.

Get the credentials

As a Microsoft Dataverse administrator, perform the following steps:

  1. Ensure you have a Dataverse Environment to work with, and you have access to that environment through https://admin.powerplatform.microsoft.com/ (https://admin.powerplatform.microsoft.com/).

  2. Ensure that you have an application registered in portal.azure.com. This application must have access to the tenant we have our Dataverse Environment available.

  3. Generate and store ClientID and Client Secret within that application.

  4. Go to Power Apps Admin Center and configure your Dataverse Environment to be accessed via applications registered before. To do that, go to Settings » Users & permissions » application users. Previously created applications must be added and granted with privileges necessary to read data from Microsoft Dataverse.

  5. Copy and save the Environment URL of the selected Dataverse Environment from https://admin.powerplatform.microsoft.com/ (https://admin.powerplatform.microsoft.com/).

Set up Snowflake account

As a Snowflake account administrator, perform the following tasks:

  1. Create a new role or use an existing role and grant the Database privileges.

  2. Create a new Snowflake service user with the type as SERVICE.

  3. Grant the Snowflake service user the role you created in the previous steps.

  4. Configure with key-pair auth for the Snowflake SERVICE user from step 2.

  5. Snowflake strongly recommends this step. Configure a secrets manager supported by Openflow, for example, AWS, Azure, and Hashicorp, and store the public and private keys in the secret store.

    Note

    If for any reason, you do not wish to use a secrets manager, then you are responsible for safeguarding the public key and private key files used for key-pair authentication according to the security policies of your organization.

    1. Once the secrets manager is configured, determine how you will authenticate to it. On AWS, it’s recommended that you the EC2 instance role associated with Openflow as this way no other secrets have to be persisted.

    2. In Openflow, configure a Parameter Provider associated with this Secrets Manager, from the hamburger menu in the upper right. Navigate to Controller Settings » Parameter Provider and then fetch your parameter values.

    3. At this point all credentials can be referenced with the associated parameter paths and no sensitive values need to be persisted within Openflow.

  6. If any other Snowflake users require access to the raw ingested documents and tables ingested by the connector (for example, for custom processing in Snowflake), then grant those users the role created in step 1.

  7. Designate a warehouse for the connector to use. Start with the smallest warehouse size, then experiment with size depending on the number of tables being replicated, and the amount of data transferred. Large table numbers typically scale better with multi-cluster warehouses, rather than larger warehouse sizes.

Configure the connector

As a data engineer, perform the following tasks to configure a connector:

  1. Create a database and schema in Snowflake for the connector to store ingested data.

  2. Download the connector definition file.

  3. Import the connector definition into Openflow:

    1. Open the Snowflake Openflow canvas.

    2. Add a process group. To do this, drag and drop the Process Group icon from the tool palette at the top of the page onto the canvas. Once you release your pointer, a Create Process Group dialog appears.

    3. On the Create Process Group dialog, select the connector definition file to import.

    4. Connect the Upload failure output of the group to your NiFi monitoring component, for example, LogAttribure.

  4. Right-click on the imported process group and select Parameters.

  5. Populate the required parameter values as described in Flow parameters.

Flow parameters

Parameter

Description

Authentication Strategy

Defines how the connector will connect to Snowflake. Use the value KEY_PAIR to specify that a private key must be provided for authentication. When running the connector on SPCS, use SNOWFLAKE_SESSION_TOKEN.

Environment URL

The main identifier of a source system to fetch data. The URL indicates a namespace where Dataverse tables exist. It also lets you create a scope parameter for OAuth.

Microsoft Azure Tenant ID

Microsoft tenant ID lets you create OAuth URLs.

Client ID

Microsoft Dataverse Web API (https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/overview) uses OAuth authentication to secure access, and the connector uses the client credentials flow. To learn about client ID and how to find it in Microsoft Entra, see Application ID (client ID) (https://learn.microsoft.com/en-us/azure/healthcare-apis/register-application#application-id-client-id).

Client Secret

Microsoft Dataverse Web API (https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/overview) uses OAuth authentication to secure access, and the connector uses the client credentials flow. To learn about client secret and how to find it in Microsoft Entra, see Certificates & secrets (https://learn.microsoft.com/en-us/azure/healthcare-apis/register-application#certificates--secrets).

Scheduling Interval

The processor that is fetching a list of tables to be ingested must be triggered according to a schedule. The interval is provided by the user.

Table Selection Strategy

Allows selection of tables based on names or regex.

Table Names

Comma separated list of tables. This is an optional filter that limits ingestion to the predefined list.

Table Name Pattern

Regex that lets you filter tables.

Snowflake Account

Name of the Snowflake account to which the connection is to be made. The format is [organization-name]-[account-name].

Destination Database

Name of the database where data is to be stored.

Destination Schema

Schema name where sink tables are to be created.

Snowflake Private Key

The RSA private key used for authentication. The RSA key must be formatted according to PKCS8 standards and have standard PEM headers and footers. Note that either Snowflake Private Key File or Snowflake Private Key must be defined.

Snowflake Private Key File

The file that contains the RSA private key used for authentication to Snowflake, which is formatted according to PKCS8 standards and has standard PEM headers and footers. The header line starts with -----BEGIN PRIVATE. Select the Reference asset checkbox to upload the private key file.

Snowflake Private Key Password

The password associated with the Snowflake Private Key File.

Run the flow

  1. Right-click on the plane and select Enable all Controller Services.

  2. Right-click on the imported process group and select Start. The connector starts the data ingestion.

Language: English