Apache Iceberg tables with Snowpark Connect for Spark¶

Snowpark Connect for Spark supports reading and writing Apache Iceberg™ tables through the standard Spark DataFrame read/write and Spark SQL APIs. It supports working with both Snowflake-managed and externally managed Iceberg tables. For general information about Iceberg tables in Snowflake, see Apache Iceberg™ tables.

Snowpark Connect for Spark executes the standard Spark/Iceberg DataFrame and SQL APIs on Snowflake using Spark Connect. Your Spark client code stays the same; Snowflake becomes the execution engine.

Iceberg table types¶

Snowpark Connect for Spark supports working with both Snowflake-managed and externally managed Iceberg tables.

Not every feature is available on every table type. The matrix in each section calls out supported deployments.

Install Iceberg SQL plugin¶

pip install 'snowpark-connect[iceberg]'

This installs Iceberg Spark SQL extension JARs. This plugin is required to use Iceberg SQL such as time-travel parsing on the Spark client. Snowpark Connect for Spark executes natively on Snowflake.

Session parameters¶

The following parameters are required on the Snowpark Connect for Spark session to enable Iceberg features to work end-to-end:

Client / session (Snowpark Connect for Spark)

Working with Snowflake-managed Iceberg tables¶

A managed Iceberg table uses Snowflake as the Iceberg catalog.

Step 1: You can choose where the table data and metadata files are stored:

• Snowflake storage: Snowflake stores and manages the Iceberg table files for you, so you don’t need to configure or grant access to external cloud storage.
• External cloud storage that you manage, which Snowflake accesses using an external volume.

To use Snowflake-managed Iceberg tables with external cloud storage, configure an external volume and link it to your Spark session in one of the following ways:

• Set the EXTERNAL_VOLUME property on the database.
• Set snowpark.connect.iceberg.external_volume in your Spark configuration:

spark.conf.set("snowpark.connect.iceberg.external_volume", "<your_volume>")

Step 2: Once you have set up an external volume, you can read and write to Snowflake-managed Iceberg tables as shown in the examples below.

Example — reading from a Snowflake-managed table

df = spark.read.format("iceberg").load("my_iceberg_table")

df = spark.read.table("my_iceberg_table")

The argument to load() is a Snowflake table identifier (for example, my_database.my_schema.my_table), not a file path or catalog URI.

Example — writing to a Snowflake-managed table using DataFrameWriter V1

df.write.format("iceberg").saveAsTable("my_iceberg_table")

Example — writing to a Snowflake-managed table using Spark SQL

INSERT INTO prod.db.events SELECT * FROM staging.raw_events;

Working with externally managed Iceberg tables¶

You need to create a catalog-linked database to work with externally managed tables. A catalog-linked database (CLD) is a Snowflake database connected to an external Iceberg REST catalog. With a catalog-linked database, you can access multiple remote Iceberg tables from Snowflake without creating individual externally managed tables.

What it does: Read and write Iceberg tables in Snowflake databases linked to external catalogs (Unity Catalog, AWS Glue). Snowpark Connect for Spark maps Spark format("iceberg") reads/writes and DataFrameWriterV2 to Snowflake CREATE ICEBERG TABLE / DML against the linked catalog.

Iceberg reference: Spark configuration — catalogs (https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs)

To use externally managed Iceberg tables with Snowpark Connect for Spark, do the following:

Step 1: Configure an external volume for your cloud storage, and then create a catalog integration for your remote Iceberg catalog.

Alternatively, if your Iceberg catalog uses vended credentials, you do not need to create an external volume. You can directly create a catalog integration using vended credentials to your remote Iceberg catalog.

Step 2: Create a catalog-linked database.

Step 3: Once a catalog-linked database has been created, you can use it for reads and writes to externally managed Iceberg tables using Snowpark Connect for Spark. In the examples below, cldglue is the catalog-linked database for Glue.

Example — reading a Glue CLD table:

df = spark.read.format("iceberg").load("cldglue.my_schema.sales")

Example — writing to a Glue CLD table using DataFrameWriter V1:

df.write.format("iceberg").saveAsTable("cldglue.my_schema.sales")

Example — create a Glue CLD table using DataFrameWriter V2:

(
    source_df.writeTo("cldglue.my_schema.new_table")
    .using("iceberg")
    .create()
)

Prerequisites: CLD database and schema provisioned in Snowflake; grants on the linked catalog schema.

DataFrameWriterV2 — Iceberg writes¶

Spark reference: DataFrameWriterV2 (https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameWriterV2.html)

In addition to DataFrameWriter V1 and Spark SQL APIs, Snowpark Connect for Spark supports writing using the new DataFrameWriterV2 API. The V2 API handles Iceberg create, replace, append, and conditional overwrite — including partition transforms and table properties — without explicit SQL DDL.

Iceberg reference: Spark writes (https://iceberg.apache.org/docs/latest/spark-writes/)

Supported operations¶

Infer Iceberg targets in CLD¶

For existing CLD Iceberg tables, append, overwrite, overwritePartitions, and replace no longer require .using("iceberg"). Snowpark Connect for Spark infers the Iceberg path from the catalog-linked target (matching Spark Connect behavior).

Create modes still require an explicit provider:

# Existing CLD table — .using("iceberg") optional
(
    new_rows.writeTo("cldglue.my_schema.sales")
    .append()
)

# New CLD table — provider required
(
    new_rows.writeTo("cldglue.my_schema.sales_new")
    .using("iceberg")
    .create()
)

CLD create / replace with correct identifier casing¶

Glue and Unity catalogs are case-insensitive. Snowpark Connect for Spark now emits explicit CREATE ICEBERG TABLE DDL (not quoted mixed-case column names) for create, createOrReplace, and replace on CLD targets. createOrReplace / replace drop the existing table first because CLDs reject CREATE OR REPLACE for externally managed Iceberg tables.

(
    df.writeTo("cldglue.my_schema.events")
    .using("iceberg")
    .partitionedBy("days(event_ts)")
    .createOrReplace()
)

partitionedBy with Iceberg transforms¶

Supports Iceberg partition transforms in partitionedBy(): years, months, days, hours, bucket.

(
    df.writeTo("my_db.my_schema.events")
    .using("iceberg")
    .partitionedBy(
        "days(event_date)",
    )
    .create()
)

overwrite(condition) — conditional row overwrite¶

Row-level conditional overwrite via DELETE WHERE … + INSERT:

(
    df.writeTo("my_db.my_schema.events")
    .overwrite(F.col("region") == "US")
)

Works on both Iceberg and standard Snowflake tables.

overwritePartitions()¶

Dynamic partition-level overwrite for Iceberg tables for transformed partitions (identity, days, hours, months, years).

(
    df.writeTo("my_db.my_schema.events")
    .overwritePartitions()
)

Only the partitions present in the incoming DataFrame are replaced in the target Iceberg table; all other partitions are preserved.

Iceberg table properties: comment, format-version, max-snapshot-age¶

tableProperty("comment", …), tableProperty("format-version", "2"), and tableProperty("max-snapshot-age.ms", "86400000") are forwarded into CREATE ICEBERG TABLE DDL.

(
    df.writeTo("my_db.my_schema.events")
    .using("iceberg")
    .tableProperty("comment", "Daily events")
    .tableProperty("format-version", "2")
    .tableProperty("max-snapshot-age.ms", "86400000")  # ms
    .create()
)

Glue CLD LOCATION / BASE_LOCATION¶

When creating Iceberg tables in Glue-backed CLDs, pass the S3 prefix via Spark’s LOCATION table property. Snowpark Connect for Spark maps it to Snowflake BASE_LOCATION:

(
    df.writeTo("cldglue.my_schema.events")
    .using("iceberg")
    .tableProperty("location", "s3china://my-bucket/prefix/events/")
    .create()
)

Iceberg write options: merge-schema¶

Pass mergeSchema or (merge-schema / merge_schema) to evolve the Iceberg table schema during a V2 overwrite() or partitionOverwrite():

(
    df.writeTo("my_db.my_schema.events")
    .option("merge-schema", "true")
    .overwrite()
)

Previously only supported for append(). CLD-backed Iceberg tables (for example, Glue) are supported.

Iceberg write options: check-ordering, check-nullability¶

Per-write validation for Iceberg V2 writes, passed via .option(...) on append, overwrite, or overwritePartitions.

• check-ordering (default true) — verifies the data’s column order matches the target table.
• check-nullability (default follows snowpark.connect.nullability.trackColumns, i.e. off) — blocks writing a nullable column into a NOT NULL column.

(
    df.writeTo("cldglue.my_schema.events")
    .option("check-ordering", "false")
    .option("check-nullability", "false")
    .append()
)

Notes:

• Iceberg only — both options are ignored on standard Snowflake tables.
• For check-nullability enforcement, set snowpark.connect.nullability.trackColumns=true or pass check-nullability=true explicitly.

Time travel¶

Snapshot-id based time travel¶

What it does: Read an Iceberg table as it existed at a specific snapshot ID. Maps to Snowflake AT(VERSION => <snapshot_id>).

Iceberg reference: Spark queries — time travel (https://iceberg.apache.org/docs/latest/spark-queries/#time-travel)

Example:

df = (
    spark.read
    .format("iceberg")
    .option("snapshot-id", 5129038471029384756)
    .load("my_db.my_schema.my_table")
)

Aliases accepted: snapshot_id, numeric versionAsOf.

Snowflake SQL generated:

SELECT * FROM my_db.my_schema.my_table
  AT(VERSION => 5129038471029384756);

Timestamp-based time travel¶

(DataFrame as-of-timestamp; 1.30.0 for SQL TIMESTAMP AS OF.)

What it does: Read the Iceberg snapshot that was current at a wall-clock timestamp. Maps to Snowflake AT(TIMESTAMP => ...).

Iceberg reference: Spark queries — time travel (https://iceberg.apache.org/docs/latest/spark-queries/#time-travel)

Example — DataFrame:

import time

# millis since Unix epoch
ts_millis = int(time.time() * 1000)

df = (
    spark.read
    .format("iceberg")
    .option("as-of-timestamp", ts_millis)
    .load("my_db.my_schema.my_table")
)

Example — Spark SQL (requires Iceberg SQL extensions on the session):

spark.conf.set(
    "spark.sql.extensions",
    "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
)

spark.sql(
    "SELECT * FROM my_db.my_schema.my_table "
    "TIMESTAMP AS OF '2026-06-15 10:30:00'"
).show()

CLD setup (one-time per table):

ALTER ICEBERG TABLE cldglue.my_schema.my_table
  SET TIME_TRAVEL_TIMESTAMP_SOURCE = 'ICEBERG_METADATA';

Version tag-based time travel¶

Tags can be used for retaining important historical snapshots for auditing purposes. Read about tags in the Iceberg branching documentation (https://iceberg.apache.org/docs/latest/branching/#historical-tags).

Snowpark Connect for Spark version 1.30.0 (tag reads); 1.31.0 (tag DDL mapping).

What it does: Read an Iceberg table at a named version tag. Maps to Snowflake AT(VERSION_TAG => '<name>').

Iceberg reference: Spark queries — time travel (https://iceberg.apache.org/docs/latest/spark-queries/#time-travel) (tags are a named ref type)

Example — DataFrame:

df = (
    spark.read
    .format("iceberg")
    .option("tag", "release_v1")
    .load("my_db.my_schema.my_table")
)

Example — Spark SQL:

spark.sql(
    "SELECT * FROM my_db.my_schema.my_table VERSION AS OF 'release_v1'"
).show()

Incremental reads¶

To read appended data incrementally, use:

• start-snapshot-id — Start snapshot ID used in incremental scans (exclusive).
• end-snapshot-id — End snapshot ID used in incremental scans (inclusive).

Snowpark Connect for Spark 1.34.0.

What it does: Read only rows appended between two Iceberg snapshots (CDC style). Maps to Snowflake CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => ...) [END (VERSION => ...)].

Iceberg reference: Spark queries — incremental read (https://iceberg.apache.org/docs/latest/spark-queries/#incremental-read)

Example:

df = (
    spark.read
    .format("iceberg")
    .option("start-snapshot-id", 1001)
    .option("end-snapshot-id", 1005)  # optional — omit for "from S1 to head"
    .load("my_db.my_schema.my_table")
)
df.show()

Managed table setup:

CREATE ICEBERG TABLE my_db.my_schema.events (
  id INT, payload STRING
) CHANGE_TRACKING = TRUE;

Support matrix (summary)¶

Version requirements (all features)¶

To use every feature in this guide, install:

Install example:

pip install 'snowpark-connect[iceberg]>=1.35.0'
# snowflake-snowpark-python>=1.53.1 is pulled in as a dependency

Known limitations¶

Most read and DataFrameWriterV2 write paths are closed as of Snowpark Connect for Spark version 1.35.0. The items below are the main customer-visible limitations still worth knowing.

Prefer DataFrameWriterV2 over Spark SQL DDL on CLD¶

Iceberg metadata sub-tables (.snapshots, .files, .manifests, .refs)¶

.snapshots, .files, .manifests, and .refs are supported on Managed, Glue CLD, Unity CLD, and Horizon. Other sub-tables (.entries, .all_manifests, etc.) return UNSUPPORTED_OPERATION.

Supported: .files, .snapshots, .manifests, .refs, .history, .metadata_log_entries

Unsupported: .all_data_files, .all_delete_files, .all_entries, .all_manifests, .delete_files, .entries, .partitions, .position_deletes

Iceberg incremental and changelog reads¶

Reading catalog-linked (Glue / Unity) Iceberg tables is supported, including time travel (VERSION AS OF / TIMESTAMP AS OF) and metadata tables. Incremental reads (start-snapshot-id / end-snapshot-id) and changelog/CDC reads are not supported on catalog-linked databases. They are only available on managed Iceberg tables.

For sub-tables that remain unsupported (.history, .entries, etc.): read metadata.json directly from object storage.

Iceberg reference: Inspecting tables (https://iceberg.apache.org/docs/latest/spark-queries/#inspecting-tables)

Schema evolution on CLD¶

Nested struct schema evolution (adding nested fields) on CLD Iceberg tables is not supported.

SQL time-travel — extensions gate¶

SQL VERSION AS OF and TIMESTAMP AS OF require spark.sql.extensions to include org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions. Install snowpark-connect[iceberg] and set the config on the Snowpark Connect for Spark server. Snowpark Connect for Spark implements execution natively on Snowflake; the config is the customer-visible support contract.

Tag writes¶

Format version 3¶

Iceberg format-version V3 on Unity CLD is not fully functional. Use format-version 2 unless your account team confirms V3 support for your catalog type.