为 Google Cloud Storage 配置外部卷

Grant Snowflake restricted access to a Google Cloud Storage (GCS) bucket using an external volume. To configure an external volume for Google Cloud Storage, you can use SQL or use Snowsight.

先决条件

在配置外部卷之前,您需要具备以下条件:

  • Google Cloud Storage 桶。

  • Google Cloud 中用于创建和管理 IAM 策略和角色的权限。如果您不是 Google Cloud 管理员,请让 Google Cloud 管理员执行这些任务。

To configure an external volume, you can use SQL or Snowsight:

Configure an external volume by using SQL

第 1 步:在 Snowflake 中创建外部卷

使用 CREATE EXTERNAL VOLUME 命令创建外部卷。

备注

只有账户管理员(具有 ACCOUNTADMIN 角色的用户)可执行此 SQL 命令。

以下示例创建一个外部卷,该卷定义了单个 GCS 加密存储位置:

CREATE EXTERNAL VOLUME my_gcs_external_volume
  STORAGE_LOCATIONS =
    (
      (
        NAME = 'my-us-west-2'
        STORAGE_PROVIDER = 'GCS'
        STORAGE_BASE_URL = 'gcs://mybucket1/path1/'
        ENCRYPTION=(TYPE='GCS_SSE_KMS' KMS_KEY_ID = '1234abcd-12ab-34cd-56ef-1234567890ab')
      )
    );
Copy

第 2 步:检索 Snowflake 账户的 GCS 服务账户

要检索为您的 Snowflake 账户自动创建的 GCS 服务账户的 ID,请使用 DESCRIBE EXTERNAL VOLUME 命令。指定先前创建的外部卷的名称。

例如:

DESC EXTERNAL VOLUME my_gcs_external_volume;
Copy

记录输出中 STORAGE_GCP_SERVICE_ACCOUNT 属性的值(例如 service-account-id@project1-123456.iam.gserviceaccount.com)。

Snowflake 会为您的整个 Snowflake 账户预置一个 GCS 服务账户。所有 GCS 外部卷都使用该服务账户。

第 3 步:授予服务账户访问桶对象的权限

在此步骤中,您要在 Google Cloud console 中配置 Snowflake 的 IAM 访问权限。

创建自定义 IAM 角色

创建具有访问桶和获取对象所需权限的自定义角色。

  1. 以项目编辑者身份登录 Google Cloud console。

  2. From the home dashboard, select IAM & Admin » Roles.

  3. 选择 Create Role

  4. 为自定义角色输入 Title 和 :extui:`Description`(可选)。

  5. 选择 Add Permissions

  6. Filter 中,选择 Service,然后选择 storage

  7. 筛选权限列表,并从列表中添加以下内容:

    • storage.buckets.get

    • storage.objects.create

    • storage.objects.delete

    • storage.objects.get

    • storage.objects.list

  8. 选择 Add

  9. 选择 Create

为 GCS 服务账户分配自定义角色

  1. 以项目编辑者身份登录 Google Cloud console。

  2. From the home dashboard, select Cloud Storage » Buckets.

  3. 筛选桶列表,然后选择创建外部卷时指定的桶。

  4. Select Permissions » View by principals, then select Grant access.

  5. Add principals 下,粘贴 第 2 步:检索 Snowflake 账户的 GCS 服务账户 的输出中的服务账户名称。

  6. Assign roles 下,选择先前创建的自定义 IAM 角色,然后选择 Save

授予 GCS 服务账户对 Google Cloud Key Management Service 密钥的权限

备注

仅当 您的 GCS 桶使用存储在 Google Cloud Key Management Service (Cloud KMS) 中的密钥进行加密时,才需要执行此步骤。

  1. 以项目编辑者身份登录 Google Cloud console。

  2. From the home dashboard, search for and select Security » Key Management.

  3. 选择分配给 GCS 桶的密钥环。

  4. 选择右上角的 SHOW INFO PANEL。此时会显示密钥环的信息面板。

  5. Add members 字段中,搜索 第 2 步:检索 Snowflake 账户的 GCS 服务账户 的 DESCRIBE EXTERNAL VOLUME 输出中的服务账户名称。

  6. Select a role 下拉列表中,选择 Cloud KMS CryptoKey Encrypter/Decrypter 角色。

  7. Select Add. The service account name is added to the Cloud KMS CryptoKey Encrypter/Decrypter role drop-down in the information panel.

第 4 步:验证存储访问

要检查 Snowflake 是否能成功对您的存储提供商进行身份验证,请调用 SYSTEM$VERIFY_EXTERNAL_VOLUME 函数。

SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME('my_external_volume');
Copy

备注

如果您收到以下错误,您的账户管理员必须在 Snowflake 部署区域中激活 AWS STS。有关说明,请参阅 AWS 文档中的 在 AWS 区域中管理 AWS STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html)。

Error assuming AWS_ROLE:
STS is not activated in this region for account:<external volume id>. Your account administrator can activate STS in this region using the IAM Console.

Configure an external volume in Snowsight

Step 1: Retrieve the GCS service account for your Snowflake account

  1. Sign in to Snowsight.

  2. In the lower-left corner, select your name » Switch role, and then select ACCOUNTADMIN or a role that has the CREATE EXTERNAL VOLUME privilege.

    For more information, see 切换您的主要角色.

  3. In the navigation menu, select Catalog » External data.

  4. Select the External volumes tab.

  5. Select + Create.

  6. Select Google Cloud Storage and then select Next.

  7. From the Grant storage access page, copy the value of the GCS service account into a text editor.

    Snowflake 会为您的整个 Snowflake 账户预置一个 GCS 服务账户。所有 GCS 外部卷都使用该服务账户。

Step 2: Grant the service account permissions to access bucket objects

在此步骤中,您要在 Google Cloud console 中配置 Snowflake 的 IAM 访问权限。

创建自定义 IAM 角色

创建具有访问桶和获取对象所需权限的自定义角色。

  1. 以项目编辑者身份登录 Google Cloud console。

  2. From the home dashboard, select IAM & Admin » Roles.

  3. 选择 Create Role

  4. 为自定义角色输入 Title 和 :extui:`Description`(可选)。

  5. 选择 Add Permissions

  6. Filter 中,选择 Service,然后选择 storage

  7. 筛选权限列表,并从列表中添加以下内容:

    • storage.buckets.get

    • storage.objects.create

    • storage.objects.delete

    • storage.objects.get

    • storage.objects.list

  8. 选择 Add

  9. 选择 Create

为 GCS 服务账户分配自定义角色

  1. 以项目编辑者身份登录 Google Cloud console。

  2. From the home dashboard, select Cloud Storage » Buckets.

  3. 筛选桶列表,然后选择创建外部卷时指定的桶。

  4. Select Permissions » View by principals, then select Grant access.

  5. Under Add principals, paste the name of the service account name from the output in Step 1: Retrieve the GCS service account for your Snowflake account.

  6. Assign roles 下,选择先前创建的自定义 IAM 角色,然后选择 Save

Step 3: Create an external volume

To create an external volume in Snowflake by using Snowsight, follow these steps:

  1. Sign in to Snowsight.

  2. In the lower-left corner, select your name » Switch role, and then select ACCOUNTADMIN or a role that has the CREATE EXTERNAL VOLUME privilege.

    For instructions, see 切换您的主要角色.

  3. In the navigation menu, select Catalog » External data.

  4. Select the External volumes tab.

  5. Select + Create.

  6. Select Google Cloud Storage and then select Next.

  7. Select Next.

  8. To configure your external volume, from the Configure external volume page, complete the fields:

    Field

    Description

    External volume name

    Enter a name for your external volume.

    Storage base URL

    Specifies the base URL for your cloud storage location.

    Encryption (optional)

    Specifies the encryption type used. Possible values are:

    • None (default): No encryption.

    • SSE-KMS (enter key): Server-side encryption using keys stored in KMS. For more information, see customer-managed encryption keys (https://cloud.google.com/storage/docs/encryption/customer-managed-keys).

    Access scope

    Specifies whether write operations are allowed for the external volume; must be set to Allow writes for the following tables:

    • Iceberg tables that use Snowflake as the catalog.

    • Iceberg tables that use an external catalog and are writable. Externally managed Iceberg tables are writable when you access them through a catalog-linked database that has the ALLOWED_WRITE_OPERATIONS parameter set to TRUE.

    For Iceberg tables created from Delta table files, setting this parameter to Allow writes enables Snowflake to write Iceberg metadata to your external storage. For more information, see 基于 Delta 的表.

    The value of this field must also match the permissions that you set on the cloud storage account for each specified storage location.

    备注

    If you plan to use the external volume for reading externally managed Iceberg tables, you can set this field to Off. Snowflake doesn't write data or Iceberg metadata files to your cloud storage when you read tables in an external Iceberg catalog.

    Scope

    Choose where this external volume should become the default location for future Iceberg tables. Possible values are:

    • Do not set a default: Don't set the external volume as a default anywhere.

    • Account: Set the external volume as the default for Iceberg tables that are created under the entire account.

    • Specific database: Set the external volume as the default for Iceberg tables that are created under the database you specify. To specify this database, use the Database drop-down that appears when you select Specific database.

    • Specific schema: Set the external volume as the default for Iceberg tables that are created under the schema you specify. To specify this schema, use the Database drop-down that appears to first select the parent database of the schema and then select the schema.

    Comment (optional)

    Specifies a comment for the external volume.

    Connectivity

    Specifies whether to use outbound private connectivity to harden your security posture. For information about using this parameter, see Google Cloud 外部存储卷的专用连接. Possible values are:

    • Public (default): Use the public internet.

    • Private (Private Service Connect): Use outbound private connectivity.

  9. Select Next.

    On the Verify connection & create volume page, Snowflake verifies your connection to Google Cloud Storage and then displays a "Successfully connected" message.

    备注

    If Snowflake is unable to verify your connection, check your permission or external volume configuration and then select Verify again.

  10. Select Create.

后续步骤

配置外部卷后,您可以创建 Iceberg 表。