Storage for Apache Iceberg™ tables¶

External volumes¶

An external volume is a named, account-level Snowflake object that you use to connect Snowflake to your external cloud storage for Iceberg tables. An external volume stores an identity and access management (IAM) entity for your storage location. Snowflake uses the IAM entity to securely connect to your storage for accessing table data, Iceberg metadata, and manifest files that store the table schema, partitions, and other metadata.

A single external volume can support one or more Iceberg tables.

Each external volume is associated with a particular Active storage location, and a single external volume can support multiple Iceberg tables. However, the number of external volumes you need depends on how you want to store, organize, and secure your table data.

You can use a single external volume if you want the data and metadata for all of your Snowflake-Iceberg tables in subdirectories under the same storage location (for example, in the same S3 bucket). To configure these directories for Snowflake-managed tables, see Data and metadata directories.

Alternatively, you can create multiple external volumes to secure various storage locations differently. For example, you might create the following external volumes:

• A read-only external volume for externally managed Iceberg tables.

• An external volume configured with read and write access for Snowflake-managed tables.

Granting Snowflake access to your storage¶

Active storage location¶

Each external volume supports a single active storage location. If you specify multiple storage locations in a CREATE EXTERNAL VOLUME statement, Snowflake assigns one location as the active location. The active location remains the same for the lifetime of the external volume.

Assignment occurs the first time you use the external volume in a CREATE ICEBERG TABLE statement. Snowflake uses the following logic to choose an active location:

• If the STORAGE_LOCATIONS list contains one or more local storage locations, Snowflake uses the first local storage location in the list. A local storage location is one with the same cloud provider and in the same region as your Snowflake account.

• If the STORAGE_LOCATIONS list does not contain any local storage locations, Snowflake selects the first location in the list.

Verifying storage access¶

To check that Snowflake can successfully authenticate to your storage provider, call the SYSTEM$VERIFY_EXTERNAL_VOLUME function.

SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME('my_s3_external_volume');

For Snowflake-managed tables, Snowflake automatically verifies access to the active storage location on your external volume in the following situations:

• The first time you specify that external volume in a CREATE ICEBERG TABLE statement for a Snowflake-managed table.

• The first time you convert a table to use Snowflake as the Iceberg catalog.

The ALLOW_WRITES property of the external volume must be set to TRUE.

Snowflake tries the following storage operations to verify the storage location.

1. Writing a test file.

2. Reading the file.

3. Listing the contents of the file’s path.

4. Deleting the file.

If any one of the operations fails, the CREATE ICEBERG TABLE (or ALTER ICEBERG TABLE … CONVERT TO MANAGED) statement fails and you receive an error message.

File management¶

This section explains how management of Iceberg table files in storage works, according to the type of Iceberg table.

CREATE OR REPLACE ICEBERG TABLE iceberg_table_1 (
  col_1 int,
  col_2 string
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'iceberg_external_volume'
  BASE_LOCATION = 'iceberg_table_1';

CREATE OR REPLACE ICEBERG TABLE iceberg_table_2 (
  col_1 int,
  col_2 string
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'iceberg_external_volume'
  BASE_LOCATION = 'iceberg_table_2';

STORAGE_BASE_URL
|-- iceberg_table_1
|   |-- data/
|   |-- metadata/
|-- iceberg_table_2
|   |-- data/
|   |-- metadata/

Enabling storage access logs¶

To diagnose issues and audit access to the storage locations associated with an external volume, you can enable storage logging. Storage logs help you identify the cause of missing or corrupted files.

Enable logging with your storage provider. Because you own and manage storage for Iceberg tables, Snowflake can’t enable logging or auditing on your Iceberg storage locations.

To learn about storage access logs for your storage provider, see the following external topics:

• Logging options for Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html)

• Usage logs and storage logs for Google Cloud Storage (https://cloud.google.com/storage/docs/access-logs)

• Azure Storage analytics logging (https://learn.microsoft.com/en-us/azure/storage/common/storage-analytics-logging)

Protecting files with versioning and object retention¶

If your Iceberg table data is in a central data repository (or data lake) that is operated on by multiple tools and services, accidental deletion or corruption might occur. To protect Iceberg table data and ensure retrieval of accidentally deleted or overwritten data, use storage lifecycle management and versioning offered by your storage provider.

With lifecycle management, you can set retention and tracking rules for storage objects. To learn about lifecycle management for your storage provider, see the following external topics:

• Managing your storage lifecycle for Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)

• Object Lifecycle Management for Google Cloud Storage (https://cloud.google.com/storage/docs/lifecycle)

• Lifecycle management policies in Azure (https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview)

To support object recovery, you can also enable versioning for your external cloud storage.

• To enable versioning for Amazon S3, see Enabling versioning on buckets (https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html).

• To enable versioning for Google Cloud Storage, see Use Object Versioning (https://cloud.google.com/storage/docs/using-object-versioning).

• To enable versioning for Azure, see Enable blob versioning (https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-enable?tabs=portal#enable-blob-versioning).

Encrypting table files¶

Snowflake can read Iceberg table files in storage that you encrypt using common server-side encryption (SSE) schemes. You should use your cloud service provider to manage encryption keys, and grant the Snowflake principal access to your keys if you use a customer-managed key.

For Amazon S3, Snowflake supports the following SSE options:

For Google Cloud Storage, Snowflake supports the following SSE option: