Metadata and retention for Apache Iceberg™ tables¶

Creation¶

Snowflake generates metadata for version 2 of the Apache Iceberg specification on a periodic basis, and writes the metadata to files on your external volume. Each new metadata file contains all DML or DDL changes since the last Snowflake-generated metadata file was created.

You can also create metadata on demand by using the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function. For instructions, see Generate snapshots of DML changes.

For information about locating metadata files, see Data and metadata directories.

Viewing metadata creation history¶

To access a full history of metadata generation attempts, view the query history for your account and filter the results. Search for the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function name in the SQL text.

Snowflake internally uses the same SYSTEM$GET_ICEBERG_TABLE_INFORMATION function to generate table metadata. Attempts made by Snowflake appear under the user called SYSTEM in the query history. The STATUS column in the query history indicates whether metadata was successfully generated.

For viewing options, see Monitor query activity with Query History.

Deletion¶

Snowflake deletes Iceberg metadata from your external cloud storage when the following events occur:

• After you drop a table.

• When the Iceberg metadata refers to snapshots or table data that has expired.

Deletion doesn’t occur immediately after the data retention period expires. As a result, metadata storage might incur costs with your cloud storage provider for longer than a table’s lifetime.

Delta-based tables¶

For Iceberg tables created from Delta table files, Snowflake automatically writes Iceberg metadata to your external storage if you configure your external volume with write access (see ALLOW_WRITES). For more information about the write location, see Data and metadata directories.

To prevent Snowflake from writing Iceberg metadata, you can set the ALLOW_WRITES parameter to FALSE on your external volume as long as no Snowflake-managed Iceberg tables use the same external volume.

Partitioning support matrix¶

The following table shows which features and actions are supported for each type of partitioned Iceberg table, and indicates compliance with version 2 of the Apache Iceberg specification.

Partitioning considerations¶

Consider the following before you use partitioned writes for Iceberg tables:

• If you use an external engine to add, drop, or replace a partition field in an externally managed table, Snowflake writes data according to the latest partition specification.

• Snowflake uses a target file size of 16 MB for partitioned tables. If the default file size is larger, Snowflake automatically decreases the target size to 16 MB when you write to a partitioned table.

• The GET_DDL function doesn’t include the PARTITION BY clause in its output.

• The sum of the sizes of the outputs for all partition transforms can’t exceed 1024 bytes for a single row.

• Because partition evolution isn’t supported for Snowflake-managed tables, you must drop the table and create a new one with partitioning.

• The DAY(), MONTH(), YEAR() partition transform parameters, which you specify within the PARTITION BY clause under table properties, are part of the Iceberg specification. For multiple days, months, or years, the partition expression parameter returns a partition for each calendar day, month, or year. For example, when the DAY() transform is used on a timestamp column that has 2 months of data, 61 partitions are created.

  In contrast, the DAY(), MONTH(), YEAR() functions in Snowflake are part of the SQL standard. For multiple days, months, or years, these functions extract the corresponding day, month, or year part from a date or timestamp. For example, when the DAY() function is used on a timestamp column that has multiple months of data, this function returns a day of the month ranging from 1 to 31.