Monitor connectors using the Openflow Connectors Dashboard¶

Filtering connectors¶

The Openflow Connectors Dashboard supports the following filters:

Event table

The Openflow connectors event table you want to monitor. This filter displays event tables that are associated with at least one Openflow deployment, as well as the default event table and the account event table. You can select only one event table at a time. Event table views are also supported.

The event table is set when you set up Openflow.

Tip

To view the event table associated with an Openflow deployment, use the DESCRIBE OPENFLOW DATA PLANE INTEGRATION command. See Set up Openflow - Snowflake Deployment or Set up Openflow - BYOC for more information on configuring event tables.

Time frame

Use this filter to identify relevant connectors in a specific time frame.

Tip

To get the most up-to-date results about the connector health, select the Last Hour time period.

Status

Enables filtering for Healthy, Unhealthy, or All connectors.

Source

Enables filtering by the source system based on known deployed connectors. The filter only shows sources that are used by your connectors.

Deployment

Enables filtering by Snowflake Openflow deployments.

This filter displays data plane integration names, which are composed of the prefix OPENFLOW_DATAPLANE_ followed by the deployment ID. To find the deployment ID, navigate to Openflow, select the Deployments tab, then select View Details.

Runtime

Enables filtering by Snowflake Openflow runtimes.

This filter displays the runtime keys. To match runtime keys with Openflow runtime names in the UI, navigate to Openflow, select the Runtimes tab, then select View Details, and find the corresponding key.

Type

Enables filtering by connector type: Databases, SaaS, Streaming, Unstructured, Other.

Change data capture connectors¶

The details page shows the following information for each table that is part of the change data capture configuration:

Table replication status

Tables can either be in Active or Failed replication status. The replication status is based on the most recent telemetry event that is available for the table. Events that cause replication to fail for a table immediately result in a Failed replication status in the dashboard. Use the Failure Reason message to identify the issue.

Error distribution

Helps you understand when the connector experienced issues, so that you can identify any potential problems with source systems, connector configuration, or the Snowflake destination.

Table name

Shows the schema and table names for all tables that are configured to be replicated by the connector. The list matches the Included Table Names or Included Table Regex configuration parameters of the connector.

Replication status

Shows whether each table is in Active or Failed replication status.

Replication phase

Shows the current table replication phase. After configuration in the connector, tables enter the New replication phase, progress to the Snapshot Load phase, perform the initial load, and ultimately enter the Incremental Replication phase when individual change data capture events are processed.

Last Ingested

Shows the timestamp of the last inserted record into the destination table during the selected time frame. When looking at this metric, consider a short delay between the records being ingested and events being logged and available to query.

You can use the Replication status, Replication phase, and time frame filters to narrow down the table list.

All connectors¶

Connector status

Shows the connector health status: Healthy if no error messages were encountered during the selected time frame, or Unhealthy if any error messages were encountered.

Error distribution

Shows a count of how many errors this connector encountered during the selected time period.

Average throughput

Measures the rate at which data is read from source systems and ingested into Snowflake for the selected connector.

• The Average throughput » Ingested metric measures how fast the selected connector ingests data into Snowflake.

• The Average throughput » Read metric measures how fast the selected connector reads data from source systems.

Total data ingested

Shows how much data the selected connector has ingested into Snowflake during the selected time period. Use this metric to quickly identify ingestion anomalies over a specific time period.

Custom flows¶

Custom flows built on the Openflow canvas can also be monitored on the dashboard, but only if they are actively version-controlled in a customer Git repository using the Openflow Git integration. Flows that aren’t version-controlled don’t appear in the dashboard.

For more information, see Version control for custom flows.

Viewing the connector errors¶

To view all errors that a connector encountered in the selected time frame, first navigate to the connector details page by selecting » View Details, and then select the Issues tab.

The error headline tells you what type of error the connector encountered, and the content provides the entire stacktrace of the error.

Viewing the connector logs¶

You might also want to look at additional connector logs to understand the context around an error message. To view all logs for the selected connector, select » View logs.

After you open the log explorer, you can also change the filters to view logs for different connectors or for entire runtimes or deployments. The log explorer supports Openflow-specific filters like the dataplane ID, the runtime key, and the process group ID.

Accessing the Openflow canvas¶

When you identify a connector issue, you probably need to navigate to the Openflow canvas to fix it; for example, adjust some configuration parameters or upgrade to a newer connector version.

To navigate to the selected connector in the Openflow canvas, select » Go to canvas.

Select a larger warehouse¶

Use the warehouse selector in the top right section of the screen to choose a different warehouse to run the queries.

Set up clustering on the Openflow event table¶

By using clustering keys, you can avoid unnecessary scanning of micro-partitions during querying, significantly accelerating the performance of queries that reference these columns. For more information, see What is Data Clustering?.

Run the following query, replacing the placeholders with your Openflow event table:

ALTER TABLE <database>.<schema>.<event_table_name>
  CLUSTER BY (
    DATE_TRUNC('HOUR', timestamp),
    RECORD_TYPE,
    CAST(record_attributes:"metricNameHash" AS STRING)
  );

Reduce the queried time frame¶

Selecting a smaller time frame in the filter scans less data and leads to faster query performance. Use the Last Hour filter for the best performance and the most up-to-date view of your connector health and performance.