Automated directory table metadata refreshes¶
You can automatically refresh the metadata for a directory table by using the following event notification services:
Amazon S3: Amazon SQS (Simple Queue Service) (https://aws.amazon.com/sqs/)
Google Cloud Storage: Google Cloud Pub/Sub (https://cloud.google.com/storage/docs/reporting-changes)
Microsoft Azure: Microsoft Azure Event Grid (https://azure.microsoft.com/en-us/services/event-grid/)
The refresh operation synchronizes the metadata with the latest set of associated files in storage, and occurs in response to the following types of changes:
New files in the path are added to the table metadata.
Files in the path are updated in the table metadata.
Files no longer in the path are removed from the table metadata.
To set up automated refreshes, see the topic for the cloud storage service where your files are located:
Refreshing directory tables automatically for Google Cloud Storage
Refreshing directory tables automatically for Azure Blob Storage
Cross-cloud support¶
Snowflake supports cross-cloud, cross-region automated directory table refreshes.
The following table shows the cross-cloud options that Snowflake supports for automated directory table refreshes, based on the cloud platform that hosts your Snowflake account.
Amazon S3 |
Google Cloud Storage |
Microsoft Azure Blob storage |
Microsoft Data Lake Storage Gen2 |
Microsoft Azure General-purpose v2 |
|
|---|---|---|---|---|---|
Accounts hosted on AWS |
✔ |
✔ |
✔ |
✔ |
✔ |
Accounts hosted on GCP |
✔ |
✔ |
✔ |
✔ |
✔ |
Accounts hosted on Azure |
✔ |
✔ |
✔ |
✔ |
✔ |
Considerations¶
Automated refreshes are event-based and provide better performance that manual refreshes for large or fast-growing stages.
Snowflake doesn’t support refreshing the directory table metadata on an internal stage. You must manually refresh the directory table metadata for an internal stage. For instructions, see Manual refresh.
Next Topics: