snowflake::ingest::streaming::StreamingIngestChannel Class Referencefinal

Snowflake Streaming Ingest Channel, which is used to ingest data into Snowflake using the Snowflake Ingest SDK. More...

#include <snowflake/ingest/streaming/channel.h>

Public Member Functions

 ~StreamingIngestChannel ()
 Close the channel if it is still open.
 StreamingIngestChannel (const StreamingIngestChannel &)=delete
StreamingIngestChannel & operator= (const StreamingIngestChannel &)=delete
Result<void> AppendRows (std::string_view ndjson,int64_t num_rows,std::optional<std::string_view> start_offset_token=std::nullopt,std::optional<std::string_view> end_offset_token=std::nullopt)
 Insert a batch of rows into the channel.
Result<void> AppendRow (std::string_view serialized_row,std::optional<std::string_view> offset_token=std::nullopt)
 Insert one row into the channel.
Result<std::future<Result<void>>> WaitForCommit (std::function<bool(const std::optional<std::string> &)> token_checker,std::chrono::milliseconds timeout=std::chrono::seconds(60)) const
 Asynchronously waits for offset token to be committed in the snowflake server side by checking whether the latest committed offset token meets the commit condition provided by the token_checker.
Result<void> Close (bool wait_for_flush=true,std::chrono::milliseconds timeout=std::chrono::milliseconds(0))
 Close the channel.
Result<ChannelStatus> GetChannelStatus () const
 Get the channel status for the current channel from Snowflake.
Result<std::optional<std::string>> GetLatestCommittedOffsetToken () const
 Get the latest committed offset token for the current channel in Snowflake.
Result<void> InitiateFlush ()
 Initiates a flush of all buffered data maintained for this Channel but does not wait for the flush to complete.
Result<std::future<Result<void>>> WaitForFlush (std::chrono::milliseconds timeout=std::chrono::milliseconds(0))
 Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side.
bool IsClosed () const
 Check if the channel is closed.
const std::string & channel_name () const noexcept
 Get the channel name.
const std::string & db_name () const noexcept
 Get the database name.
const std::string & schema_name () const noexcept
 Get the schema name.
const std::string & pipe_name () const noexcept
 Get the pipe name.

Friends

class StreamingIngestClient

Detailed Description

Snowflake Streaming Ingest Channel, which is used to ingest data into Snowflake using the Snowflake Ingest SDK.

The channel is created by calling StreamingIngestClient::OpenChannel() and closed by calling StreamingIngestChannel::Close(). Not copyable or movable.

Warning
Thread safety: Concurrent AppendRow/ AppendRows on the same (non-elastic) channel are discouraged and may lead to data loss. All other operations on this object must complete before it is destroyed, except a WaitForFlush or WaitForCommit future.

Constructor & Destructor Documentation

~StreamingIngestChannel

StreamingIngestChannel::~StreamingIngestChannel()

Close the channel if it is still open.

It waits for all the data in this channel to be flushed to the snowflake server side, for at most 60 seconds. Any error is ignored.

Warning
There is no guarantee that all the data in this channel is flushed to the snowflake server side: if the flush fails or the timeout is reached, the channel is closed without the remaining data being flushed. To be sure all data is flushed, or to get the error if any or use a different timeout, call StreamingIngestChannel::Close before the channel is destroyed.

Member Function Documentation

AppendRow

Result<void>
StreamingIngestChannel::AppendRow(
std::string_view serialized_row,
std::optional<std::string_view> offset_token = std::nullopt ))

Insert one row into the channel.

Returns once the row is queued in the SDK's in-memory buffer.

Parameters
serialized_rowdata to write: a caller-serialized JSON-formatted string that maps Column Name to Column Value for the columns in the table, e.g. R"({"c1": 1, "c2": "abc"})". The string should:
  • hold exactly one row.
  • encode BINARY values following the pipe's BINARY_INPUT_FORMAT.
  • not be modified concurrently while this call is in progress.
offset_tokenoffset of given row, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nullopt if you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details.
Returns
an empty Result on success, or a StreamingIngestError on failure.

AppendRows

Result<void>
StreamingIngestChannel::AppendRows(
std::string_view ndjson,
int64_t num_rows,
std::optional<std::string_view> start_offset_token = std::nullopt,
std::optional<std::string_view> end_offset_token = std::nullopt ))

Insert a batch of rows into the channel.

Returns once the rows are queued in the SDK's in-memory buffer.

Parameters
ndjsondata to write: a single string holding one or more rows of JSON, with rows separated by a newline character (\n). Each row is a caller-serialized JSON object that maps Column Name to Column Value for the columns in the table. For example, for two rows: "{\"c1\":1,\"c2\":\"abc\"}\\n{\"c1\":2,\"c2\":\"def\"}". The string should:
  • hold exactly num_rows rows
  • encode BINARY values following the pipe's BINARY_INPUT_FORMAT.
  • not be modified concurrently while this call is in progress.
num_rowsnumber of rows in ndjson.
start_offset_tokenstart offset of the batch/row-set, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nullopt if you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details.
end_offset_tokenend offset of the batch/row-set, used to track the ingestion progress and replay ingestion in case of failures. It could be std::nullopt if you don't plan on replaying or can't replay. Please see https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens for more details.
Returns
an empty Result on success, or a StreamingIngestError on failure.

channel_name

const std::string & StreamingIngestChannel::channel_name() const
inlinenoexcept

Get the channel name.

Returns
the channel name

Close

Result<void>
StreamingIngestChannel::Close(
bool wait_for_flush = true,
std::chrono::milliseconds timeout = std::chrono::milliseconds(0) ))

Close the channel.

If wait_for_flush=true this function will make sure all the data in this channel is flushed to the snowflake server side or return a timeout error if the timeout is reached.

Parameters
wait_for_flushwhether to wait for all data to be flushed
timeoutThe maximum time (non-negative) for close to wait. If 0, the operation will wait indefinitely.
Returns
an empty Result on success, or a StreamingIngestError on failure.

db_name

const std::string & StreamingIngestChannel::db_name() const
inlinenoexcept

Get the database name.

Returns
name of the database

GetChannelStatus

Result<ChannelStatus> StreamingIngestChannel::GetChannelStatus() const

Get the channel status for the current channel from Snowflake.

Returns
the channel status on success, or a StreamingIngestError on failure.

GetLatestCommittedOffsetToken

Result<std::optional<std::string>>
StreamingIngestChannel::GetLatestCommittedOffsetToken() const

Get the latest committed offset token for the current channel in Snowflake.

Returns
the latest committed offset token, or std::nullopt if there is none yet. Returns a StreamingIngestError directly in case of error.

InitiateFlush

Result<void> StreamingIngestChannel::InitiateFlush()

Initiates a flush of all buffered data maintained for this Channel but does not wait for the flush to complete.

Calls to AppendRow/AppendRows are still allowed on the Channel after invoking this API.

This method triggers an immediate flush of all currently buffered data in this specific channel, similar to the client-level StreamingIngestClient::InitiateFlush() but scoped to only this channel. The flush operation will occur asynchronously and this method returns immediately.

Usage: This method is useful when you want to force immediate transmission of buffered data without waiting for automatic flush triggers (time-based or size-based). It provides fine-grained control over when data gets sent to Snowflake on a per-channel basis. However, calling InitiateFlush at a high rate will lead to a drop in overall throughput, potential increase in costs, and could lead to higher incidence of throttling by the Snowflake Service.

Returns
an empty Result on success, or a StreamingIngestError on failure.

IsClosed

bool StreamingIngestChannel::IsClosed() const

Check if the channel is closed.

Returns
a boolean which indicates whether the channel is closed

pipe_name

const std::string & StreamingIngestChannel::pipe_name() const
inlinenoexcept

Get the pipe name.

Returns
name of the pipe

schema_name

const std::string & StreamingIngestChannel::schema_name() const
inlinenoexcept

Get the schema name.

Returns
name of the schema

WaitForCommit

Result<std::future<Result<void>>>
StreamingIngestChannel::WaitForCommit(
std::function<bool(const std::optional<std::string> &)> token_checker,
std::chrono::milliseconds timeout = std::chrono::seconds(60) ) const)

Asynchronously waits for offset token to be committed in the snowflake server side by checking whether the latest committed offset token meets the commit condition provided by the token_checker.

Note that snowflake commits offset token in batch, so the token_checker should be able to handle the case where the latest committed offset token passed the expected ones. That said, the token_checker usually does a range check whether the provided token is greater or equal to the expected one, not an exact match.

Dropping the returned future does not stop the polling. The wait ends when the token is committed, the timeout expires, the channel is closed or invalidated, or any error occurs.

Parameters
token_checkerA predicate that tests whether the current committed offset token from the server meets the desired condition. The predicate receives the latest committed offset token (which may be std::nullopt) and should return true when the wait condition is satisfied.
timeoutThe maximum time (non-negative) to wait for the condition to be met. A timeout of 0 checks once without retrying.
Returns
a std::future that completes with an empty Result if the commit condition was satisfied within the timeout period, or with kWaitForCommitTimeout if the timeout was reached before the condition was satisfied. Returns a StreamingIngestError directly in case of error.

WaitForFlush

Result<std::future<Result<void>>>
StreamingIngestChannel::WaitForFlush(
std::chrono::milliseconds timeout = std::chrono::milliseconds(0))

Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side.

This method triggers a flush of all pending data and waits for the flush operation to complete.

Parameters
timeoutThe maximum time (non-negative) to wait for the flush to complete. If 0, the operation will wait indefinitely.
Returns
a std::future that completes with an empty Result when the flush is done, or with kChannelWaitForFlushTimeout if the timeout is reached. Returns a StreamingIngestError directly in case of error.