Configurations for Snowpipe Streaming with high-performance architecture¶

This guide describes the configuration settings for the high-performance Snowpipe Streaming client that are available in the Java, Python, and Node.js SDKs. There are two distinct kinds of configuration:

• Process-wide environment variables: Variables that control logging and metrics for the entire running application and must be set before the client is initialized.
• Client-side properties: Properties that define the secure connection and ingestion target — such as, url, user, and private_key — and are configured for a specific client object, typically through an inline map or a profile.json file.

A single application can run multiple client objects. Each object has its own client-side properties, but they all share the same process-wide environment variable settings for logging and metrics.

The high-performance architecture requires the client to be explicitly bound to a specific PIPE object, which manages the schema, transformations, and ingestion into the target table.

Environment variables¶

These configuration settings control process-wide behavior like logging and metrics collection and must be configured as environment variables before the client object is initialized. The following table shows the environmental variables that apply to all Snowpipe Streaming client objects within the same process:

Required properties¶

The high-performance SDK mandates several properties to establish both the secure connection and the specific ingestion target (the PIPE). The following table shows the required connection and user authentication properties:

If authorization_type is set to JWT, which is the default, you must provide either the key content or the key file path, as shown in the following table:

Optional properties¶

The following table shows the high-performance SDK optional properties:

Externalizing secrets¶

Snowflake strongly recommends that you externalize secrets, such as the private_key and OAuth credentials, and store them in a key management service; for example, AWS KMS.

Configuration examples¶

The following examples show client-side and environment variable configurations.

Client-side configuration through a profile.json file¶

The following example shows how to define client-side properties:

// profile.json
{
  "authorization_type": "JWT",
  "url": "https://<account_identifier>.snowflakecomputing.cn",
  "user": "MY_SNOWFLAKE_USER",
  "account": "XY12345",
  "private_key_file": "/path/to/rsa_key.p8",
  "role": "MY_INGEST_ROLE"
}

Client-side configuration provided inline¶

The following examples show how to define client-side properties directly in code:

config = {
    "authorization_type": "JWT",
    "url": "https://<account_identifier>.snowflakecomputing.cn",
    "user": "MY_SNOWFLAKE_USER",
    "account": "XY12345",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
}
# ... code to initialize client with 'config'

const { createClient } = require("snowpipe-streaming");

const client = await createClient({
  clientName: "my-client",
  dbName: "MY_DATABASE",
  schemaName: "MY_SCHEMA",
  pipeName: "MY_PIPE",
  properties: {
    authorization_type: "JWT",
    url: "https://<account_identifier>.snowflakecomputing.cn",
    user: "MY_SNOWFLAKE_USER",
    account: "XY12345",
    private_key: "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
  },
});

Environment variable configuration¶

The following examples show how to define process-wide environment variables in the shell before you run the application:

Linux or macOS (Bash or Zsh)¶

# Set the log level for the entire application process to 'warn'
export SS_LOG_LEVEL=warn

# Change the IP for metrics to a specific loopback address
export SS_METRICS_IP=127.0.0.5

# Now run your application

Windows (command prompt)¶

# Set the log level for the entire application process to 'warn'
set SS_LOG_LEVEL=warn

# Change the metrics port
set SS_METRICS_PORT=55000

# Now run your application