Parquet Direct¶
Overview¶
Parquet Direct lets you create a read-only Apache Iceberg™ table in Snowflake from Apache Parquet™ files that you manage in object storage. Snowflake registers your existing Parquet files in place, so you can query your data lake without copying the data or rewriting it into another table format.
Parquet Direct is one type of Direct Iceberg table. Use it when your data lake contains plain Parquet files, with or without Hive-style partitioning, that aren’t managed by a table format such as Iceberg or Delta Lake.
Parquet Direct offers the following benefits:
- Performance: Queries use Snowflake’s Iceberg table engine, which delivers better performance than external tables.
- Cost: Snowflake queries Parquet files in-place without any additional storage costs.
- Seamless syncing: Snowflake can automatically refresh tables to reflect files that are added, changed, or deleted in storage, including schema changes.
- Hive-style partitioning: Snowflake can automatically infer partition columns from Hive-style
key=valuepaths to simplify table creation.
Note
Parquet Direct tables are read-only. You can’t perform DML operations (INSERT, UPDATE, DELETE) on these tables in Snowflake. Instead, use other engines to modify the files in cloud storage, and refresh the table to pick up the changes. For all limitations, see Considerations and limitations.
How Parquet Direct works¶
A Parquet Direct table uses three objects:
- An external volume that gives Snowflake read access to the storage location that contains your Parquet files.
- A catalog integration with
CATALOG_SOURCE = OBJECT_STOREand aTABLE_FORMATofNONEorHIVE. The table format tells Snowflake how to interpret your directory layout. - An Iceberg table, created with CREATE ICEBERG TABLE (Parquet files in object storage), that points to a base location on the external volume.
When you create the table and each time you refresh it, Snowflake lists the Parquet files under the base location and registers new, changed, and deleted files in the table’s metadata. Queries read the registered files directly from your storage.
Partitioning¶
To improve query performance, Snowflake strongly recommends that you partition tables created from Parquet files. Query response time is faster when Snowflake processes only a small part of the data instead of scanning the entire dataset. To partition a table, your data must be organized using logical paths.
How partition columns work depends on the TABLE_FORMAT of the catalog integration:
TABLE_FORMAT = HIVE: Snowflake infers partition column names and values from Hive-stylekey=valuedirectory names in your file paths (for example,year=2024/month=01/). You don’t write partition expressions, and you can’t define explicit partition columns.TABLE_FORMAT = NONE: Use when your paths aren’t Hive-style. You define partition columns explicitly by using expressions that parse theMETADATA$FILENAMEpseudocolumn; for example, a regular expression that extracts a date or identifier from the path. For an example, see Define a partition column with an expression.
In both cases, Snowflake computes partitions when the table is refreshed.
If your data lake has Hive-style partition keys with inconsistent casing (for example, some files under Year=2024/ and
others under year=2024/), set MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE on the table so that Snowflake resolves all
casing variants to the same partition column.
Create a Parquet Direct table¶
To create a Parquet Direct table, complete the following steps. For complete, end-to-end instructions that also set up auto-refresh, see the tutorials.
Step 1: Create an external volume¶
To create an external volume, complete the instructions for your cloud storage service:
Step 2: Create a catalog integration¶
Create a catalog integration by using the CREATE CATALOG INTEGRATION (Object storage) command with
CATALOG_SOURCE = OBJECT_STORE. Choose the table format that matches your file layout:
Step 3: Create an Iceberg table¶
Create a table by using the CREATE ICEBERG TABLE (Parquet files in object storage) command. The following example lets Snowflake infer the schema and partition columns, and evolve the schema on later refreshes:
Schema evolution is disabled by default, and you can’t enable it after you create the table, so include
ENABLE_SCHEMA_EVOLUTION = TRUE in the CREATE ICEBERG TABLE statement.
For more examples, including explicit column definitions and partition expressions, see Examples.
Refresh a Parquet Direct table¶
Refreshing a table registers the latest changes to your Parquet files in object storage. You can refresh the table automatically when Snowflake receives event notifications from your cloud storage, or refresh it manually. Snowflake recommends auto-refresh.
Auto-refresh¶
Parquet Direct supports event-driven auto-refresh for all three major cloud storage providers. Auto-refresh is serverless: Snowflake processes event notifications for new, changed, and deleted files as they arrive, without a warehouse.
To set up auto-refresh, follow the tutorial for your cloud storage provider:
- Tutorial: Auto-refresh a Parquet Direct table from Amazon S3
- Tutorial: Auto-refresh a Parquet Direct table from Azure Blob Storage
- Tutorial: Auto-refresh a Parquet Direct table from Google Cloud Storage
When you enable auto-refresh on a table that previously had auto-refresh disabled, Snowflake queues notifications for files that were added while auto-refresh was off and processes a full backfill refresh asynchronously.
Manual refresh¶
To manually refresh a table, use ALTER ICEBERG TABLE … REFRESH. You can refresh the entire base location, or specify a path relative to the base location to refresh a directory, a partition, or a single file:
Note
- Snowflake matches the path as a prefix. For example,
'/year=2024/month=1'refreshes bothmonth=1andmonth=10throughmonth=12. To refresh only one directory, end the path with a slash, as in'/year=2024/month=1/'. For more examples, see ALTER ICEBERG TABLE … REFRESH. - If you specify a path that doesn’t exist, the refresh proceeds as if you didn’t specify a path.
- You can’t manually refresh a table that has auto-refresh enabled. To manually refresh the table, set
AUTO_REFRESH = FALSEfirst.
Schema evolution¶
With ENABLE_SCHEMA_EVOLUTION = TRUE, a refresh synchronizes the table with the following schema changes in your Parquet
files:
- New columns
- Type widening, as defined by the Apache Iceberg specification, for the following types:
| Original type | Widened type | Notes |
|---|---|---|
| int | long | NUMBER(10,0) to NUMBER(19,0) |
| float | double | Both map to Snowflake FLOAT |
| decimal(p1, s) | decimal(p2, s) | Precision can increase; for example, decimal(5,2) to decimal(7,2) |
Type widening also applies recursively to struct fields, array elements, and map keys and values. For example:
Schema evolution is disabled by default, even when Snowflake infers the schema at creation time. You can only enable schema evolution when you create the table:
After you create the table, you can disable schema evolution, but you can’t enable it again:
Without schema evolution, you can’t change a Parquet Direct table’s schema. To pick up schema changes in an existing table
that was created without schema evolution, re-create the table with ENABLE_SCHEMA_EVOLUTION = TRUE.
Best practices¶
Use schema inference and schema evolution¶
Let Snowflake infer the table schema and evolve it over time:
- Schema inference reads your Parquet files and chooses the most appropriate Snowflake data type for each column.
- Schema evolution ensures that new columns and type widening in your files aren’t silently missed on refresh.
When you omit the column list in CREATE ICEBERG TABLE, Snowflake enables schema inference by default, but not schema
evolution. Set ENABLE_SCHEMA_EVOLUTION = TRUE when you create the table, because you can’t enable schema evolution
later.
Use Hive partition inference¶
If your files use Hive-style key=value paths, create the catalog integration with TABLE_FORMAT = HIVE so that
Snowflake infers partition columns automatically, instead of writing and maintaining partition expressions.
Use auto-refresh¶
Auto-refresh is event-driven and serverless. Compared with scheduled manual refreshes, it’s usually less expensive and picks up incremental changes much faster, because Snowflake processes only the files named in event notifications instead of listing the entire base location.
Use auto-refresh when creating large tables¶
By default, CREATE ICEBERG TABLE registers all existing files before it completes, which can take a long time for a large
table. You can use auto-refresh to register the existing files instead. Create the table with both
REFRESH_ON_CREATE = FALSE and AUTO_REFRESH = FALSE, then turn on auto-refresh:
Snowflake then registers the existing files in the background. Instead of blocking until every file is registered, the table is queryable immediately and returns more results as files are registered.
Note
When you use REFRESH_ON_CREATE = FALSE, you must define the table’s columns explicitly, because schema inference requires
files at creation time.
Register only the files you need from cold storage¶
If your Parquet files are in a cold or archive storage tier, avoid registering every file in the base location, because
registration reads each file and requires it to be in a hot tier. Instead, create the table with
REFRESH_ON_CREATE = FALSE, then manually refresh only the paths
that your queries need. Snowflake reads only the files under those paths, so you only need to move those files to a hot
tier:
Snowflake doesn’t register files when you query the table. Queries return data only from the files that you registered.
Give each table its own storage folder¶
Keep each table’s files in their own folder, and don’t point one table’s BASE_LOCATION inside another table’s base
location. Disjoint folders keep each table’s contents unambiguous, and ensure that any storage credentials scoped to a
table, such as with future credential vending, aren’t over-scoped to the files of other tables.
The following layout is recommended. Each table has its own folder:
Avoid the following layout. The RETURNS table’s folder is nested inside the folder for the ORDERS table, so ORDERS also picks up the RETURNS files:
Monitor refresh¶
Snowflake provides the following tools to monitor Parquet Direct tables:
| Tool | Best used for |
|---|---|
| Event tables | Debugging auto-refresh at the file level, and building real-time alerts and notifications. |
| SYSTEM$PARQUET_DIRECT_AUTO_REFRESH_STATUS | A current snapshot of auto-refresh health for a table. |
| ICEBERG_TABLE_FILE_REGISTRATION_HISTORY and ICEBERG_TABLE_FILES | Which files were registered, when, and which files the table currently contains. |
| PIPE_USAGE_HISTORY view | Auto-refresh history and compute costs over time. |
Monitor auto-refresh with an event table¶
Snowflake can record auto-refresh events for Parquet Direct tables in your active event table. Events are emitted for auto-refresh only; manual refreshes and table creation don’t emit events.
Note
To use an event table to monitor Parquet Direct auto-refresh, contact Snowflake Support to request access.
Log levels¶
To capture events, set the LOG_EVENT_LEVEL parameter on the table, or on its schema, database, or account. Snowflake captures events at the level you set and at more severe levels:
| Level | Event name | What it captures |
|---|---|---|
ERROR | parquet_direct_file_ingestion_failure | One event for each file that auto-refresh failed to register. Body fields: file_path, error_message. |
INFO | parquet_direct_auto_refresh_batch_success | One event for each successfully committed batch of files. Body fields: add_count, delete_count, skip_count, ingested_size (bytes), total_latency, processing_latency. |
DEBUG | parquet_direct_file_lifecycle | One event for each file that auto-refresh registered or removed. Body fields: file_path, state (INGESTED or DELETED). |
Latency fields are in milliseconds. Parquet Direct tables don’t emit WARN events.
Events use the scope snow.ingest.parquet_direct. The resource_attributes column includes the database, schema, and table
name.
For most tables, set LOG_EVENT_LEVEL = ERROR to capture failures, or INFO to also track successful refreshes. Use DEBUG
only while you troubleshoot specific files, because it records an event for every file.
Query auto-refresh events¶
The following example retrieves recent auto-refresh events for a table:
The event body fields are in the VALUE column.
The following example summarizes files added, deleted, and bytes registered by auto-refresh per hour:
The following example lists files that failed to register in the last day:
Alert on auto-refresh failures¶
You can create an alert that notifies you when auto-refresh fails to register files. The following example checks for failure events every 10 minutes and sends an email by using an existing email notification integration:
You can integrate Snowflake notifications with external systems such as Slack, Microsoft Teams, PagerDuty, Jira, and ServiceNow by using the webhooks.
Check auto-refresh status¶
Use SYSTEM$PARQUET_DIRECT_AUTO_REFRESH_STATUS to get a current snapshot of auto-refresh for a table, including its execution state, the number of files waiting to be processed, and the most recent notification and error from the notification channel:
To extract individual fields, parse the result:
An executionState of RUNNING means auto-refresh is healthy. STALLED means Snowflake encountered an error and is
attempting to recover, and STOPPED means auto-refresh stopped and requires action; see
Refresh fails.
Check registered files¶
Use the following Information Schema table functions to see which Parquet files Snowflake registered for a table:
- ICEBERG_TABLE_FILE_REGISTRATION_HISTORY returns the history of file registration operations, including failures and the reason for each failure.
- ICEBERG_TABLE_FILES returns the files that the table currently contains.
Track auto-refresh history and compute costs¶
Auto-refresh credits appear in the PIPE_USAGE_HISTORY view view.
Note
To query Parquet Direct tables in the PIPE_USAGE_HISTORY view, contact Snowflake Support to request access.
The following example shows daily auto-refresh credits and bytes for a Parquet Direct table over the last 30 days:
For Parquet Direct tables, the PIPE_NAME column contains the table name, without the database or schema name.
Troubleshooting¶
Refresh succeeds but data is missing¶
If a refresh succeeds but some data doesn’t appear in query results, complete the following steps:
-
Check whether the table’s
PATTERNexcludes the missing files. Use DESCRIBE ICEBERG TABLE or GET_DDL to see the table definition, and verify that the missing file paths match the regular expression. -
Check whether Snowflake registered the files, and whether any registration failed:
To see every file that the table currently contains, query ICEBERG_TABLE_FILES.
-
If the table uses auto-refresh, check when Snowflake last received a notification from your cloud storage:
If
lastReceivedMessageTimestampis older than the time you added the files andpendingFileCount= 0, Snowflake didn’t receive notifications for them. This could be because of a misconfiguration either between your bucket and queue, or your queue and Snowflake account. Verify your cloud storage event notification configuration; see the tutorial for your cloud storage provider.
Refresh fails¶
-
Find the most recent failure:
-
For auto-refresh, call SYSTEM$PARQUET_DIRECT_AUTO_REFRESH_STATUS and check the
executionState,invalidExecutionStateReason, andchannelErrorMessagefields. If you set up an event table, query it forparquet_direct_file_ingestion_failureevents. -
For a manual refresh, look up the refresh statement by its query ID in QUERY_HISTORY , QUERY_HISTORY_BY_* and check the error message:
-
-
If auto-refresh is failing, fix the underlying issue, then reset auto-refresh by running a manual refresh:
-
Disable auto-refresh on the table:
-
Manually refresh the table:
-
Re-enable auto-refresh:
-
Refresh fails with a schema error¶
If a refresh fails because a file’s schema doesn’t match the table, compare the file’s schema with the table definition:
-
Run INFER_SCHEMA on the failing file:
This step requires a stage that points to the same storage location and a Parquet file format.
-
Get the table definition with GET_DDL, and compare the column names and types:
If the file adds columns or widens types and the table was created without
schema evolution, re-create the table with
ENABLE_SCHEMA_EVOLUTION = TRUE.
If the file changes a column type in a way that schema evolution doesn’t support, fix the file or create a new table.
Initial table creation takes a long time¶
By default, CREATE ICEBERG TABLE registers all existing files in the base location before it completes. For a large table, use one of the following approaches instead:
- Create the table with
REFRESH_ON_CREATE = FALSEto skip the initial registration, then register older files gradually by manually refreshing specific paths. - Create the table with auto-refresh so the table is queryable immediately and returns more results as files are registered.
With REFRESH_ON_CREATE = FALSE, Snowflake doesn’t register files when you query the table. The table returns no rows
until you refresh it or auto-refresh registers files.
You can’t change the table schema¶
Without schema evolution, you can’t modify a Parquet Direct table’s schema, and you can’t enable schema evolution on an
existing table. To let the table pick up new columns and widened types from your files, re-create the table with
ENABLE_SCHEMA_EVOLUTION = TRUE. For more information, see Schema evolution.
Billing¶
Parquet Direct costs have the following components:
| Component | What you pay for |
|---|---|
| Table creation and manual refresh | Compute used for registering Parquet files during table creation and refresh. |
| Auto-refresh | Serverless compute to process event notifications and register files, plus data registration for the bytes added. Auto-refresh compute appears in the PIPE_USAGE_HISTORY view. |
| Data registration | 0.35 credits per terabyte of Parquet data added to a table, for every creation, manual refresh, and auto-refresh. Files that are deleted or unchanged aren’t charged. |
Important
Billing for data registration takes effect on November 1, 2026. Data registration before that date isn’t billed.
For more information about the credits charged for each service, see the Snowflake Service Consumption Table.
Query Parquet Direct costs¶
The following example returns auto-refresh compute credits for each table over the last 30 days. To query Parquet Direct tables in the PIPE_USAGE_HISTORY view, contact Snowflake Support to request access.
Data types¶
For how Parquet data types map to Snowflake data types, see Parquet data types.
Supported features¶
Parquet Direct tables support the following Snowflake features:
- Queries: Query Parquet Direct tables with SQL like any other Iceberg table, and use them as the source for CREATE TABLE … AS SELECT (CTAS) and CREATE TABLE … LIKE.
- Materialized views: Create materialized views on Parquet Direct tables.
- Secure views: Create secure views on Parquet Direct tables.
- Insert-only streams: Track files that are added to a Parquet Direct table. Standard and append-only streams aren’t supported.
- Tasks: Use Parquet Direct tables in scheduled and triggered tasks.
- Access control and data protection: Control access with role-based access control, including organization users and groups, and protect data with masking policies and row access policies.
- Performance: Use the search optimization service and the query acceleration service with Parquet Direct tables.
- Data sharing: Share Parquet Direct tables, including local sharing, cross-region sharing, and listings that use Cross-Cloud Auto-Fulfillment.
- Data types: Use semi-structured data types, including VARIANT, and geospatial data types. For how Parquet types map to Snowflake types, see Parquet data types.
Parquet Direct tables also support the following capabilities for working with Parquet files:
- Schema inference and schema evolution: Snowflake can infer column definitions from your files when you create the table, and evolve the schema on later refreshes. For more information, see Schema evolution.
- Partitioning: Snowflake infers partition columns from Hive-style
key=valuepaths when you use a catalog integration withTABLE_FORMAT = HIVE, or derives them from expressions that you define withTABLE_FORMAT = NONE. For more information, see Partitioning. - Case-insensitive column matching: Files that use different casings for a column name populate the same table
column. For more information, see the
MATCH_BY_COLUMN_NAMEparameter of CREATE ICEBERG TABLE (Parquet files in object storage). - Refresh: Keep tables in sync with your files by using auto-refresh with Amazon S3, Azure Blob Storage, or Google Cloud Storage, or by refreshing manually, optionally for a specific path.
- Monitoring: Monitor auto-refresh with event tables, SYSTEM$PARQUET_DIRECT_AUTO_REFRESH_STATUS, and ICEBERG_TABLE_FILE_REGISTRATION_HISTORY. For more information, see Monitor refresh.
For features that aren’t supported, see Considerations and limitations.
Considerations and limitations¶
-
Parquet files that use any of the following features or data types aren’t supported:
- Invalid or duplicate field IDs. Files with valid field IDs are supported; Snowflake matches columns by name, not by field ID.
- The DECIMAL data type with precision higher than 38.
- LIST or MAP types with one-level or two-level representation.
- Unsigned integer types (
INT(signed=false)). - The FLOAT16 data type.
-
Snowflake currently only supports creating Parquet Direct tables with an object storage-based catalog integration. Other catalog integration types are not supported.
-
When auto-refresh is enabled on a table, you can’t manually refresh the table. To manually refresh the table, disable auto-refresh first.
-
Parquet Direct tables are read-only:
- You can’t perform DML operations (INSERT, UPDATE, DELETE, MERGE) on these tables in Snowflake. Use other engines to modify the files in your cloud storage.
- You can’t convert these tables to Snowflake-managed Iceberg tables.
- You can’t perform table maintenance on these tables.
-
The following Snowflake features aren’t supported for Parquet Direct tables:
- Cloning
- Clustering (
CLUSTER BY) - Replication
- Standard streams and append-only streams (insert-only streams are supported)
- Dynamic tables
- Native Apps
-
You can’t create a Parquet Direct table by using CREATE TABLE … AS SELECT (CTAS) or CREATE TABLE … LIKE. You can use a Parquet Direct table as the source for either statement.
-
You can’t undrop a Parquet Direct table. After you drop a Parquet Direct table, re-create it to register the files again.
-
You can’t set the
DATA_RETENTION_TIME_IN_DAYSparameter on a Parquet Direct table. -
You can’t enable schema evolution on an existing Parquet Direct table. To use schema evolution, set
ENABLE_SCHEMA_EVOLUTION = TRUEwhen you create the table. -
Data sharing is supported for Parquet Direct tables, including local sharing, cross-region sharing, and listings that use Cross-Cloud Auto-Fulfillment.