snowflake::ingest::streaming::StreamingIngestClient Class Referencefinal

A class that is the starting point for using the Streaming Ingest client APIs, a single client maps to exactly one account/database/schema/pipe in Snowflake; however, multiple clients can point to the same account/database/schema/pipe. More...

#include <snowflake/ingest/streaming/client.h>

Public Member Functions

 ~StreamingIngestClient ()
 Close the client if it is still open.
 StreamingIngestClient (const StreamingIngestClient &)=delete
StreamingIngestClient & operator= (const StreamingIngestClient &)=delete
Result<std::shared_ptr<StreamingIngestElasticChannel>> GetElasticChannel ()
 Get the elastic channel for this client.
Result<std::pair<std::unique_ptr<StreamingIngestChannel>, ChannelStatus>> OpenChannel (std::string_view channel_name,std::optional<std::string_view> offset_token=std::nullopt)
 Open a channel with a given channel name.
Result<void> Close (bool wait_for_flush=true,std::chrono::milliseconds timeout=std::chrono::milliseconds(0))
 Close the client.
Result<void> DropChannel (std::string_view channel_name)
 Drop the specified channel on the server using the channel name.
Result<std::map<std::string, ChannelStatus>> GetChannelStatuses (const std::vector<std::string> &channel_names) const
 Return the channel status for the given list of channels from Snowflake.
Result<std::map<std::string, std::optional<std::string>>> GetLatestCommittedOffsetTokens (const std::vector<std::string> &channel_names) const
 Return the latest committed/persisted offset token for the given list of channels in Snowflake.
Result<void> InitiateFlush ()
 Initiates a flush by the Client which causes all outstanding buffered data to be flushed to Snowflake.
Result<std::future<Result<void>>> WaitForFlush (std::chrono::milliseconds timeout=std::chrono::milliseconds(0))
 Asynchronously waits for all buffered data in all channels managed by this client to be flushed to the Snowflake server side.
bool IsClosed () const
 Check if the client is closed.
const std::string & client_name () const noexcept
 Get the client name.
const std::string & db_name () const noexcept
 Get the database name.
const std::string & schema_name () const noexcept
 Get the schema name.
const std::string & pipe_name () const noexcept
 Get the pipe name.

Static Public Member Functions

static Result<std::unique_ptr<StreamingIngestClient>> Create (const ClientOptions &options)
 Create a new client.
static Result<std::unique_ptr<StreamingIngestClient>> CreateFromTable (const TableClientOptions &options)
 Create a table-mode client.

Detailed Description

A class that is the starting point for using the Streaming Ingest client APIs, a single client maps to exactly one account/database/schema/pipe in Snowflake; however, multiple clients can point to the same account/database/schema/pipe.

Each client will contain information for Snowflake authentication and authorization, and it will be used to create one or more StreamingIngestChannel.

The client is created by calling StreamingIngestClient::Create and closed by calling StreamingIngestClient::Close. Not copyable or movable.

Warning
Thread safety: All other operations on this object must complete before it is destroyed, except WaitForFlush future.

Constructor & Destructor Documentation

~StreamingIngestClient

StreamingIngestClient::~StreamingIngestClient()

Close the client if it is still open.

It waits for all the data in all channels managed by this client to be flushed to the Snowflake server side, for at most 60 seconds. Any error is ignored.

Warning
There is no guarantee that all the data is flushed to the Snowflake server side: if the flush fails or the timeout is reached, the client is closed without the remaining data being flushed. To be sure all data is flushed, or to get the error or use a different timeout, call StreamingIngestClient::Close before the client is destroyed.
Destroyed inside an elastic channel handler, the client is closed without flushing, because the flush would wait on the task running the handler; buffered rows in all its channels are lost.

Member Function Documentation

client_name

const std::string & StreamingIngestClient::client_name() const
inlinenoexcept

Get the client name.

Returns
the client name

Close

Result<void>
StreamingIngestClient::Close(
bool wait_for_flush = true,
std::chrono::milliseconds timeout = std::chrono::milliseconds(0) ))

Close the client.

If wait_for_flush=true this function will make sure all the data in all channels managed by this client is flushed to the Snowflake server side or return an error if the timeout is reached.

Parameters
wait_for_flushwhether to wait for all channels to be flushed
timeoutThe maximum time (non-negative) for close to wait. If 0, the operation will wait indefinitely.
Returns
an empty Result on success, or a StreamingIngestError on failure.

Create

Result<std::unique_ptr<StreamingIngestClient>>
StreamingIngestClient::Create(const ClientOptions & options)
static

Create a new client.

Parameters
optionsthe client configuration; see ClientOptions
Returns
the client on success, or a StreamingIngestError on failure

CreateFromTable

Result<std::unique_ptr<StreamingIngestClient>>
StreamingIngestClient::CreateFromTable(const TableClientOptions & options)
static

Create a table-mode client.

The client writes to the table's default TABLE-STREAMING pipe.

Parameters
optionsthe table-mode client configuration
Returns
the client on success, or a StreamingIngestError on failure.

db_name

const std::string & StreamingIngestClient::db_name() const
inlinenoexcept

Get the database name.

Returns
name of the database

DropChannel

Result<void>
StreamingIngestClient::DropChannel(std::string_view channel_name)

Drop the specified channel on the server using the channel name.

Note that this call will blindly drop the latest version of the channel and any pending data will be lost. It will also delete Offset Token and other state from Snowflake servers. So only use it if you are completely done ingesting data for this channel. If you open a channel with the same name in the future, it will behave like a new channel.

Also see StreamingIngestChannel::Close() to close channels. That approach will drop the local version of the channel and if the channel has been concurrently reopened by another client, that version of the channel won't be affected.

Parameters
channel_namethe channel name to drop
Returns
an empty Result on success, or a StreamingIngestError on failure.

GetChannelStatuses

Result<std::map<std::string, ChannelStatus>>
StreamingIngestClient::GetChannelStatuses(
const std::vector<std::string> & channel_names) const

Return the channel status for the given list of channels from Snowflake.

Parameters
channel_namesthe list of channel names
Returns
the ChannelStatus for each channel, keyed by channel name, or a StreamingIngestError on failure.

GetElasticChannel

Result<std::shared_ptr<StreamingIngestElasticChannel>>
StreamingIngestClient::GetElasticChannel()

Get the elastic channel for this client.

Elastic channels have no offset token concepts and their lifecycle is tied to the client. The same instance is returned on repeated calls (singleton).

Returns
the elastic channel for this client on success, or a StreamingIngestError on failure

GetLatestCommittedOffsetTokens

Result<std::map<std::string, std::optional<std::string>>>
StreamingIngestClient::GetLatestCommittedOffsetTokens(
const std::vector<std::string> & channel_names) const

Return the latest committed/persisted offset token for the given list of channels in Snowflake.

Parameters
channel_namesthe list of channel names
Returns
the latest committed offset token for each channel, keyed by channel name, or a StreamingIngestError on failure.

InitiateFlush

Result<void> StreamingIngestClient::InitiateFlush()

Initiates a flush by the Client which causes all outstanding buffered data to be flushed to Snowflake.

Note that data can still be accepted by the Client - this is an asynchronous call and will return after flush is initiated for all Channels opened by this Client

Returns
an empty Result on success, or a StreamingIngestError on failure.

IsClosed

bool StreamingIngestClient::IsClosed() const

Check if the client is closed.

Returns
a boolean which indicates whether the client is closed

OpenChannel

Result<std::pair<std::unique_ptr<StreamingIngestChannel>, ChannelStatus>>
StreamingIngestClient::OpenChannel(
std::string_view channel_name,
std::optional<std::string_view> offset_token = std::nullopt ))

Open a channel with a given channel name.

The channel is opened on the account/database/schema/pipe defined by the current client

Parameters
channel_namethe name of the channel to open
offset_tokenthe optional offset token to set on the channel to be opened. If not provided and this reopens an existing channel, the latest persisted offset token will be reused.
Returns
the channel and its ChannelStatus on success, or a StreamingIngestError on failure

pipe_name

const std::string & StreamingIngestClient::pipe_name() const
inlinenoexcept

Get the pipe name.

Returns
name of the pipe

schema_name

const std::string & StreamingIngestClient::schema_name() const
inlinenoexcept

Get the schema name.

Returns
name of the schema

WaitForFlush

Result<std::future<Result<void>>>
StreamingIngestClient::WaitForFlush(
std::chrono::milliseconds timeout = std::chrono::milliseconds(0))

Asynchronously waits for all buffered data in all channels managed by this client to be flushed to the Snowflake server side.

This method triggers a flush of all pending data across all channels and waits for the flush operations to complete.

Parameters
timeoutThe maximum time (non-negative) to wait for all channels to flush. If 0, the operation will wait indefinitely.
Returns
a std::future that completes with an empty Result when the flush is done, or with kClientWaitForFlushTimeout if the timeout is reached. Returns a StreamingIngestError directly in case of error.