snowflake::ingest::streaming::StreamingIngestChannel Class Referencefinal
Snowflake Streaming Ingest Channel, which is used to ingest data into Snowflake using the Snowflake Ingest SDK. More...
#include <snowflake/ingest/streaming/channel.h>
Public Member Functions | |
| ~StreamingIngestChannel () | |
| Close the channel if it is still open. | |
| StreamingIngestChannel (const StreamingIngestChannel &)=delete | |
| StreamingIngestChannel & | operator= (const StreamingIngestChannel &)=delete |
| Result< | AppendRows (std::string_view ndjson,int64_t num_rows,std::optional<std::string_view> start_offset_token=std::nullopt,std::optional<std::string_view> end_offset_token=std::nullopt) |
| Insert a batch of rows into the channel. | |
| Result< | AppendRow (std::string_view serialized_row,std::optional<std::string_view> offset_token=std::nullopt) |
| Insert one row into the channel. | |
| Result< | WaitForCommit (std::function<bool(const std::optional<std::string> &)> token_checker,std::chrono::milliseconds timeout=std::chrono::seconds(60)) const |
| Asynchronously waits for offset token to be committed in the snowflake server side by checking whether the latest committed offset token meets the commit condition provided by the token_checker. | |
| Result< | Close (bool wait_for_flush=true,std::chrono::milliseconds timeout=std::chrono::milliseconds(0)) |
| Close the channel. | |
| Result< | GetChannelStatus () const |
| Get the channel status for the current channel from Snowflake. | |
| Result< | GetLatestCommittedOffsetToken () const |
| Get the latest committed offset token for the current channel in Snowflake. | |
| Result< | InitiateFlush () |
| Initiates a flush of all buffered data maintained for this Channel but does not wait for the flush to complete. | |
| Result< | WaitForFlush (std::chrono::milliseconds timeout=std::chrono::milliseconds(0)) |
| Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side. | |
| bool | IsClosed () const |
| Check if the channel is closed. | |
| const std:: | channel_name () const noexcept |
| Get the channel name. | |
| const std:: | db_name () const noexcept |
| Get the database name. | |
| const std:: | schema_name () const noexcept |
| Get the schema name. | |
| const std:: | pipe_name () const noexcept |
| Get the pipe name. | |
Friends | |
| class | StreamingIngestClient |
Detailed Description
Snowflake Streaming Ingest Channel, which is used to ingest data into Snowflake using the Snowflake Ingest SDK.
The channel is created by calling StreamingIngestClient::OpenChannel() and closed by calling StreamingIngestChannel::Close(). Not copyable or movable.
- Warning
- Thread safety: Concurrent
AppendRow/AppendRowson the same (non-elastic) channel are discouraged and may lead to data loss. All other operations on this object must complete before it is destroyed, except aWaitForFlushorWaitForCommitfuture.
Constructor & Destructor Documentation
~StreamingIngestChannel¶
Close the channel if it is still open.
It waits for all the data in this channel to be flushed to the snowflake server side, for at most 60 seconds. Any error is ignored.
- Warning
- There is no guarantee that all the data in this channel is flushed to the snowflake server side: if the flush fails or the timeout is reached, the channel is closed without the remaining data being flushed. To be sure all data is flushed, or to get the error if any or use a different timeout, call
StreamingIngestChannel::Closebefore the channel is destroyed.
Member Function Documentation
AppendRow¶
StreamingIngestChannel::AppendRow(
std::string_view serialized_row,
std::optional<std::string_view> offset_token = std::nullopt ))
Insert one row into the channel.
Returns once the row is queued in the SDK's in-memory buffer.
- Parameters
-
serialized_row data to write: a caller-serialized JSON-formatted string that maps Column Name to Column Value for the columns in the table, e.g. R"({"c1": 1, "c2": "abc"})". The string should:- hold exactly one row.
- encode BINARY values following the pipe's
BINARY_INPUT_FORMAT. - not be modified concurrently while this call is in progress.
offset_token offset of given row, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nulloptif you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
AppendRows¶
StreamingIngestChannel::AppendRows(
std::string_view ndjson,
int64_t num_rows,
std::optional<std::string_view> start_offset_token = std::nullopt,
std::optional<std::string_view> end_offset_token = std::nullopt ))
Insert a batch of rows into the channel.
Returns once the rows are queued in the SDK's in-memory buffer.
- Parameters
-
ndjson data to write: a single string holding one or more rows of JSON, with rows separated by a newline character ( \n). Each row is a caller-serialized JSON object that maps Column Name to Column Value for the columns in the table. For example, for two rows:"{\"c1\":1,\"c2\":\"abc\"}\\n{\"c1\":2,\"c2\":\"def\"}". The string should:- hold exactly
num_rowsrows - encode BINARY values following the pipe's
BINARY_INPUT_FORMAT. - not be modified concurrently while this call is in progress.
num_rows number of rows in ndjson.start_offset_token start offset of the batch/row-set, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nulloptif you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details.end_offset_token end offset of the batch/row-set, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nulloptif you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details. - hold exactly
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
channel_name¶
|
const std::string & StreamingIngestChannel::channel_name() const
|
inlinenoexcept |
Get the channel name.
- Returns
- the channel name
Close¶
StreamingIngestChannel::Close(
bool wait_for_flush = true,
std::chrono::milliseconds timeout = std::chrono::milliseconds(0) ))
Close the channel.
If wait_for_flush=true this function will make sure all the data in this channel is flushed to the snowflake server side or return a timeout error if the timeout is reached.
- Parameters
-
wait_for_flush whether to wait for all data to be flushed timeout The maximum time (non-negative) for close to wait. If 0, the operation will wait indefinitely.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
db_name¶
|
const std::string & StreamingIngestChannel::db_name() const
|
inlinenoexcept |
Get the database name.
- Returns
- name of the database
GetChannelStatus¶
Get the channel status for the current channel from Snowflake.
- Returns
- the channel status on success, or a
StreamingIngestErroron failure.
GetLatestCommittedOffsetToken¶
Get the latest committed offset token for the current channel in Snowflake.
- Returns
- the latest committed offset token, or
std::nulloptif there is none yet. Returns aStreamingIngestErrordirectly in case of error.
InitiateFlush¶
Initiates a flush of all buffered data maintained for this Channel but does not wait for the flush to complete.
Calls to AppendRow/AppendRows are still allowed on the Channel after invoking this API.
This method triggers an immediate flush of all currently buffered data in this specific channel, similar to the client-level StreamingIngestClient::InitiateFlush() but scoped to only this channel. The flush operation will occur asynchronously and this method returns immediately.
Usage: This method is useful when you want to force immediate transmission of buffered data without waiting for automatic flush triggers (time-based or size-based). It provides fine-grained control over when data gets sent to Snowflake on a per-channel basis. However, calling InitiateFlush at a high rate will lead to a drop in overall throughput, potential increase in costs, and could lead to higher incidence of throttling by the Snowflake Service.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
IsClosed¶
Check if the channel is closed.
- Returns
- a boolean which indicates whether the channel is closed
pipe_name¶
|
const std::string & StreamingIngestChannel::pipe_name() const
|
inlinenoexcept |
Get the pipe name.
- Returns
- name of the pipe
schema_name¶
|
const std::string & StreamingIngestChannel::schema_name() const
|
inlinenoexcept |
Get the schema name.
- Returns
- name of the schema
WaitForCommit¶
Asynchronously waits for offset token to be committed in the snowflake server side by checking whether the latest committed offset token meets the commit condition provided by the token_checker.
Note that snowflake commits offset token in batch, so the token_checker should be able to handle the case where the latest committed offset token passed the expected ones. That said, the token_checker usually does a range check whether the provided token is greater or equal to the expected one, not an exact match.
Dropping the returned future does not stop the polling. The wait ends when the token is committed, the timeout expires, the channel is closed or invalidated, or any error occurs.
- Parameters
-
token_checker A predicate that tests whether the current committed offset token from the server meets the desired condition. The predicate receives the latest committed offset token (which may be std::nullopt) and should returntruewhen the wait condition is satisfied.timeout The maximum time (non-negative) to wait for the condition to be met. A timeout of 0checks once without retrying.
- Returns
- a
std::futurethat completes with an emptyResultif the commit condition was satisfied within the timeout period, or withkWaitForCommitTimeoutif the timeout was reached before the condition was satisfied. Returns aStreamingIngestErrordirectly in case of error.
WaitForFlush¶
Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side.
This method triggers a flush of all pending data and waits for the flush operation to complete.
- Parameters
-
timeout The maximum time (non-negative) to wait for the flush to complete. If 0, the operation will wait indefinitely.
- Returns
- a
std::futurethat completes with an emptyResultwhen the flush is done, or withkChannelWaitForFlushTimeoutif the timeout is reached. Returns aStreamingIngestErrordirectly in case of error.