Openflow Connector for Amazon Kinesis Data Streams¶
备注
此连接器受 Snowflake 连接器条款 的约束。
About¶
This topic describes the basic concepts of Openflow Connector for Amazon Kinesis Data Streams, including its workflow and limitations.
您可以使用 Amazon Kinesis Data Streams (https://docs.aws.amazon.com/streams/latest/dev/introduction.html) 实时收集和处理大量数据流记录。制作者持续不断地将数据推送到 Kinesis Data Streams，而使用者则实时处理这些数据。
Kinesis 数据流由一组 分片 (https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard) 组成。每个分片都有一系列数据记录。数据记录是存储在 Kinesis 数据流中的数据单位。数据记录由序列号、分区键和数据 blob 组成，数据 blob 是一段不可变的字节序列。
Openflow Connector for Amazon Kinesis Data Streams reads data from Kinesis streams and writes it into Snowflake tables using the Snowpipe Streaming architecture.
如果您希望实现以下功能，可以使用此连接器：
Ingest real-time events from Amazon Kinesis into Snowflake for near real-time analytics
Ingest real-time events from Amazon Kinesis into Snowflake-managed Iceberg™ tables
Accelerate your ingestion even more by combining Openflow speed with the Interactive Tables feature
Use Single Message Transforms to enrich or filter data before it lands in Snowflake.
限制¶
One connector supports only ingestion from a single stream.
The connector does not support schema evolution for Apache Iceberg™ tables.
Autoscaling is not supported. The number of Openflow runtime min and max nodes should be constant for the runtime where Openflow Connector for Amazon Kinesis Data Streams is deployed.
Limitations of fault tolerance with the connector¶
Kinesis Streams can be configured with a retention time. If for any reason the Openflow Connector for Amazon Kinesis Data Streams is not able to ingest data for more than the retention time, then expired records will not be loaded.
Supported data types and authentication methods¶
The connector by default is configured to work with the JSON data type and supports authentication using AWS Credentials: Access Key ID and Secret Access Key. Connector can be customized to work with other data types and authentication methods.