教程：开始使用 Snowpipe Streaming 高性能架构 SDK¶

先决条件¶

在运行演示之前，请确保满足以下先决条件：

• Snowflake account: Verify that you have access to a Snowflake account. You will need a user with sufficient privileges (e.g., ACCOUNTADMIN or USERADMIN for the initial setup) to create the dedicated user and custom role detailed in 第 1 步：配置 Snowflake 对象.

• Network access: Ensure that your network allows outbound connectivity to Snowflake and Amazon S3 or Google Cloud Platform (GCS) or Azure Blob Storage. Adjust firewall rules if necessary because the SDK makes REST API calls to Snowflake and to your cloud storage provider.

  • To verify network connectivity, use the following command:

  # Test connectivity to Snowflake; replace with your account URL
  curl -I https://<your_account_identifier>.snowflakecomputing.cn
  
  # Test connectivity to AWS S3
  curl -I https://s3.amazonaws.com
  
  # Test connectivity to GCS
  curl -I https://storage.googleapis.com
  
  # Test connectivity to Azure Blob Storage
  curl -I https://azure.blob.core.windows.net  or curl -I https://<your_account_name>.blob.core.windows.net
  

  Copy

• Java 开发环境：安装 Java 11 或更高版本，以及用于管理依赖关系的 Maven。

• Python：安装 Python 版本 3.9 或更高版本。

• System requirements: The SDK requires glibc version 2.26 or later. You can check your current glibc version with:

  ldd --version
  

  Copy

• Snowpipe Streaming SDKs and the sample code:

  • For AWS: Obtain the Java SDK (https://central.sonatype.com/artifact/com.snowflake/snowpipe-streaming) or Python SDK (https://pypi.org/project/snowpipe-streaming/) (any version).

  • For Azure: Requires SDK version 1.1.0 or later.

  • For GCP: Requires SDK version 1.1.0 or later.

  Download the sample code for your preferred language from the Snowpipe Streaming SDK examples in the GitHub repository (https://github.com/snowflakedb/snowpipe-streaming-sdk-examples).

开始使用¶

本节概述了设置和运行演示应用程序所需的步骤。

openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

PUBK=$(cat ./rsa_key.pub | grep -v KEY- | tr -d '\012')
echo "ALTER USER MY_USER SET RSA_PUBLIC_KEY='$PUBK';"

-- 1. Create a dedicated role and user (Run with a highly-privileged role)
CREATE OR REPLACE USER MY_USER;
CREATE ROLE IF NOT EXISTS MY_ROLE;
GRANT ROLE MY_ROLE TO USER MY_USER;

-- 2. Set the public key for key-pair authentication
-- NOTE: Replace 'YOUR_FORMATTED_PUBLIC_KEY' with the output of the PUBK variable from the key generation step.
ALTER USER MY_USER SET RSA_PUBLIC_KEY='YOUR_FORMATTED_PUBLIC_KEY';

-- 3. Set the default role (Recommended)
ALTER USER MY_USER SET DEFAULT_ROLE=MY_ROLE;

-- 4. Switch to the new role and create objects
USE ROLE MY_ROLE;
-- NOTE: You may also need to run USE WAREHOUSE YOUR_WH; here if a default warehouse isn't set.

-- Create database and schema
CREATE OR REPLACE DATABASE MY_DATABASE;
CREATE OR REPLACE SCHEMA MY_SCHEMA;

-- Create a target table
CREATE OR REPLACE TABLE MY_TABLE (
    data VARIANT,
    c1 NUMBER,
    c2 STRING
);

-- 5. Configure authentication policy (Optional, but recommended for explicit control)
CREATE OR REPLACE AUTHENTICATION POLICY testing_auth_policy
  AUTHENTICATION_METHODS = ('KEYPAIR')
  CLIENT_TYPES = ('DRIVERS');

-- Apply authentication policy (if created)
ALTER USER MY_USER SET AUTHENTICATION POLICY testing_auth_policy;

{
    "user": "MY_USER",
    "account": "your_account_identifier",
    "url": "https://your_account_identifier.snowflakecomputing.cn:443",
    "private_key_file": "rsa_key.p8",
    "role": "MY_ROLE"
}

<dependency>
    <groupId>com.snowflake</groupId>
    <artifactId>snowpipe-streaming</artifactId>
    <version>YOUR_SDK_VERSION</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.18.1</version>
</dependency>

pip install snowpipe-streaming

python example.py

SELECT COUNT(*) FROM MY_DATABASE.MY_SCHEMA.MY_TABLE;
SELECT * FROM MY_DATABASE.MY_SCHEMA.MY_TABLE LIMIT 10;