关于 Openflow Connector for Kinesis

备注

This connector is subject to the Snowflake Connector Terms.

This topic describes the basic concepts of Openflow Connector for Kinesis, including its workflow and limitations.

您可以使用 Amazon Kinesis Data Streams (https://docs.aws.amazon.com/streams/latest/dev/introduction.html) 实时收集和处理大量数据流记录。制作者持续不断地将数据推送到 Kinesis Data Streams，而使用者则实时处理这些数据。

Kinesis 数据流由一组 分片 (https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard) 组成。每个分片都有一系列数据记录。数据记录是存储在 Kinesis 数据流中的数据单位。数据记录由序列号、分区键和数据 blob 组成，数据 blob 是一段不可变的字节序列。

The Openflow Connector for Kinesis reads data from a Kinesis data stream and writes it to a Snowflake table using Snowpipe Streaming.

Use cases

Use this connector if you want to ingest real‐time events from Amazon Kinesis Data Streams into Snowflake for near real-time analytics.

工作流程

AWS administrator tasks

  1. Create credentials for the connector to connect with Kinesis Stream and the associated DynamoDB.

  2. Set up IAM policies that have the permissions listed in IAM permissions required for KCL consumer applications (https://docs.aws.amazon.com/streams/latest/dev/kcl-iam-permissions.html).

  3. Record the stream name and application name and provide them to your Snowflake account administrator. These are required when setting up the connector in the runtime.

Snowflake account administrator tasks ------------------------------------------------------------------------------------------------===

  1. Install the connector.

  2. Configure the connector:

    1. Provide the AWS and Snowflake credentials and settings.

    2. Provide the Kinesis stream name.

    3. Set the database and schema names in the Snowflake account.

    4. Customize other parameters.

  3. Run the connector in the Openflow canvas. Upon execution, the connector performs the following actions:

    1. 创建用于存储 Kinesis Stream 检查点的 DynamoDB 表。

    2. 提取流数据。

    3. 如果至少从流中接收到一条记录，则在 Snowflake 数据库中创建已配置的目标表。

    4. 将处理后的数据加载到指定的 Snowflake 表中。

Business user tasks ------------------------------------------------------------------------------------------------===

Perform operations on the data downloaded from Kinesis into the destination table.

限制

  • The connector supports only a single stream.

  • If you use a manually created table:

    • The table name must match the stream of the data it holds precisely.

    • The table name must be uppercase.

  • The connector supports only JSON message format.

  • The connector supports only Amazon Access Key IAM authentication.

  • The connector logs failed messages to the Snowflake logs and does not route them to a DLQ stream.

后续步骤

For information on how to set up the connector, see the following topic:

语言: 中文