Best practices for dbt Projects on Snowflake¶

This guide provides opinionated best practices for data engineering teams running dbt at scale on Snowflake. Each section is self-contained, so you can jump to the topic that matters most to your team.

dbt Projects on Snowflake eliminates infrastructure you’d otherwise manage yourself. There’s no Python environment to maintain, no Airflow cluster to scale, no dbt CLI version drift across developer machines. Snowflake handles the runtime, orchestration (through tasks), and dbt version management natively. This lets your team focus on transformation logic and data quality rather than infrastructure operations.

Cost optimization¶

Reducing warehouse compute time is one of the highest-impact best practices for teams running dbt at scale. The following patterns help you avoid unnecessary processing.

Use incremental models for large, frequently updated tables¶

If your source table is large and receives regular updates, use the incremental materialization instead of a full table rebuild. Incremental models scan only the rows that changed since the last run, which drastically reduces the time window processed before transformation occurs.

If no new rows exist since the last run, dbt still runs the filter query to check, but the transformation itself is effectively skipped because there’s nothing to merge or insert. This consumes a fraction of the warehouse time because only the lightweight filter query runs, not the full transformation. This makes incremental models especially cost-effective for event-driven pipelines or tables with append-only patterns.

For small tables that update infrequently, a full table materialization is simpler and often just as fast. Use incremental models where the scan reduction provides a measurable benefit.

Increase parallelism with threads¶

Configure the threads parameter in your profiles.yml to control how many models dbt runs concurrently within a single execution. To be compatible with most Snowflake warehouses, Snowflake recommends setting your threads to 8. A higher thread count than 1 allows independent models to execute in parallel, reducing total wall-clock time for a given run.

my_target:
  type: snowflake
  threads: 8
  # ...

Choose a thread count that matches your warehouse’s available compute capacity without causing queuing.

For more details about costs, see Understanding costs for dbt Projects on Snowflake.

Choosing a dbt version¶

Snowflake supports both dbt Core (Python-based, versions 1.x) and dbt Fusion (Rust-based, versions 2.x). For most teams, start with the latest supported dbt Core version in dbt Projects on Snowflake (1.11.x). It has the broadest Snowflake materialization and dbt package support.

If your project becomes very large (more than 5,000 models) or your team wants to future-proof for performance improvements, consider moving to dbt Fusion. Keep in mind:

Some migration is required when moving from Core to Fusion.
Not all dbt packages are compatible with Fusion today. Check the dbt package hub (https://hub.getdbt.com/) for Fusion-compatible badges before upgrading.
The most popular dbt package hub packages (dbt_utils, dbt_expectations, dbt_project_evaluator) are already compatible.

For the full list of supported versions and how to set account-level defaults, see Migrate to dbt Fusion. For Fusion migration guidance, see Upgrading to v2.0 (https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v2).

Orchestration¶

Snowflake tasks provide native scheduling for dbt project execution without requiring external orchestrators like Airflow. Understanding the permission model is critical to setting up orchestration correctly.

Understand the two-role model of execution¶

Every EXECUTE DBT PROJECT statement involves two roles:

The calling role — the active role of the session (or task owner role) that issues the EXECUTE DBT PROJECT statement. This role must have the EXECUTE DBT PROJECT privilege on the dbt project object.
The profiles.yml role — the role specified in your profiles.yml target. This role defines what the dbt run can actually access (databases, schemas, tables, warehouses) during execution.

Both roles must have USAGE on the warehouse. The calling role must also be able to USE the profiles.yml role. Operations during execution are restricted to the privileges that both roles have in common.

This two-role model applies whether you run dbt interactively (a human running EXECUTE DBT PROJECT in a worksheet) or through a scheduled task. The difference is:

Interactive execution: Your active session role is the calling role.
Task execution: The task runs as a system service with the privileges of the task owner role (the role that has OWNERSHIP on the task). No specific user is associated with the run.

Use a dedicated service account role¶

Have a dedicated service account (for example, github_actions_service_user) create and own your tasks. This ensures:

Task privileges are governed and auditable in one place.
The service account role can be granted only the minimum permissions needed.
Departing team members don’t break production scheduling.

Align your warehouse configuration¶

Use the same warehouse in both your task definition and your profiles.yml target to avoid waking two warehouses for a single orchestration run:

-- Task warehouse matches profiles.yml warehouse
CREATE OR ALTER TASK my_db.my_schema.run_dbt_daily
  WAREHOUSE = transform_wh
  SCHEDULE = '360 minutes'
AS
  EXECUTE DBT PROJECT my_db.my_schema.my_project args='build --target prod';

# profiles.yml
prod:
  type: snowflake
  warehouse: transform_wh
  # ...

If the task uses warehouse_a but your profiles.yml specifies warehouse_b, both warehouses wake up for one run.

Use task graphs for multi-step pipelines¶

Use the AFTER clause to chain tasks into a graph. For example, run tests after your models complete:

CREATE OR ALTER TASK my_db.my_schema.test_dbt_daily
  WAREHOUSE = transform_wh
  AFTER my_db.my_schema.run_dbt_daily
AS
  EXECUTE DBT PROJECT my_db.my_schema.my_project args='test --target prod';

For more details, see Schedule execution of dbt project objects on Snowflake.

Continuous integration and continuous deployment (CI/CD)¶

CI/CD pipelines ensure that every change to your dbt project is validated before reaching production. This is essential for teams working collaboratively on shared pipelines.

Don’t deploy to production without CI/CD¶

Teams sometimes bypass CI/CD validation by deploying directly to a production dbt project object. Common anti-patterns include:

Using the Git stage as a shortcut to deploy changes without creating a pull request.
Deploying from Workspaces by doing a git pull and then deploying via the Workspaces UI.

These approaches should be used when deploying and testing a dev/staging dbt project object. But for production, always deploy through a CI/CD pipeline. Without CI validation:

Broken models reach production undetected.
Tests never run until after data is already corrupted.
There’s no review gate for teammates to catch errors.

The recommended CI/CD pattern¶

Use the Snowflake CLI within a CI orchestrator (such as GitHub Actions) to gate every deployment behind validation:

A developer opens a pull request.
The CI pipeline triggers and creates a tester dbt project object using snow dbt deploy.
The pipeline runs snow dbt execute with dbt build against a dev or staging target.
If all models build successfully and all tests pass, the PR is mergeable.
On merge to main, a separate pipeline deploys to the production dbt project object.

This ensures that no change reaches production without passing your full test suite. For the complete setup walkthrough, see Tutorial: Set up CI/CD integrations on dbt Projects on Snowflake.

Multi-account setups¶

For teams with separate staging and production Snowflake accounts, CI/CD is the best pattern to manage different deployment destinations:

On pull request: Deploy and test against your staging account.
On merge to main: Deploy to your production account.

The Snowflake CLI supports inline connection overrides, making it straightforward to target different accounts:

# During a CI job, deploy to your staging account
snow dbt deploy my_project \
  --account my_org-staging \
  --database analytics_staging \
  --role svc_dbt_role \
  --warehouse transform_wh \
  --default-target staging

# Later, in a CD job, deploy to your production account
snow dbt deploy my_project \
  --account my_org-production \
  --database analytics_prod \
  --role svc_dbt_role \
  --warehouse transform_wh \
  --default-target prod

Enforce quality at the PR level¶

Set up branch protection rules so that pull requests can’t merge into main until all CI checks pass. This guarantees quality regardless of where the commit originated (Workspaces, local IDE, or Cortex Code Desktop).

Permissions and privileges¶

Understanding the separation between dbt project object permissions and data access is critical for teams operating dbt at scale.

MONITOR privilege doesn’t grant data access¶

The MONITOR privilege on a dbt project object gives a user access to the dbt project object’s manifest, run history, and artifacts. It does not grant access to query the tables and views that the dbt project object creates.

This means a data engineer with MONITOR can:

View model definitions, schemas, and lineage in the dbt DAG.
Review execution logs and run history.
Access dbt artifacts (manifest.json, run_results.json, dbt.log) from each execution using system functions.
Use CoCo to inspect the files of a deployed dbt project object and debug production failures.

But they cannot query the production tables unless they have SELECT privileges granted separately.

Separate data engineering from data querying¶

The permissions for running a dbt pipeline are separate from the permissions for querying its output:

Data engineering permissions: EXECUTE on the dbt project object, ownership or access to source tables in dev, ability to deploy new versions.
Data querying permissions: SELECT on the production tables and views created by the pipeline.

This separation is intentional. A data engineer might only have access to run operations in their dev schema or staging environment, while a dedicated service account runs the production pipeline. Analysts then receive access to the production output through standard role grants, not through the dbt project object itself.

Artifacts describe schema, not data¶

dbt manifests and artifacts only describe the schema of models (column names, types, relationships). They don’t expose any live data from production tables. Granting MONITOR to a broader audience for documentation and lineage purposes is safe from a data-access perspective.

For the full permissions reference, see Access control for dbt projects on Snowflake.

Security¶

Security best practices for dbt Projects on Snowflake center on minimizing credential exposure and maintaining strict access boundaries.

Use OIDC ephemeral tokens for CI/CD¶

For GitHub Actions and other CI/CD platforms, use OpenID Connect (OIDC) authentication instead of long-lived service account credentials. With OIDC:

Tokens are ephemeral and scoped to the specific workflow run.
No secrets to store, rotate, or risk leaking.
The OIDC user has only the permissions needed to deploy and execute the tester or production dbt project object.

This is a significant security improvement over storing Snowflake credentials as repository secrets. For setup details, see the OIDC section in the CI/CD tutorial.

Require multi-factor authentication for human users¶

For any human user who can deploy or execute dbt project objects, require multi-factor authentication (MFA) or personal access tokens (PATs). This protects against credential compromise and ensures that only authorized engineers can modify production pipelines.

Output tables don’t auto-grant access¶

When a dbt pipeline creates or updates tables, those tables don’t automatically become accessible to the user or role that ran the pipeline. Access to production output requires explicit GRANT statements from an administrator.

This means:

Running a pipeline doesn’t give you SELECT on the results.
Analysts need separate grants to query production tables.
Your data engineering permissions and your data querying permissions remain isolated.

This design keeps your security boundary clean: the ability to build a pipeline is separate from the ability to read its output.

Hybrid development¶

Large engineering teams have developers with different skill levels and tool preferences. dbt Projects on Snowflake supports multiple development workflows so teams can choose what works best for each member.

Development environment options¶

Environment	Best for	Key benefit
Cortex Code Desktop (Snowflake-managed mode)	Experienced developers who prefer a full IDE	Local IDE experience with Snowflake-native dbt execution
Snowflake Workspaces	Teams that want browser-based development with no local setup	Zero installation, collaborative, Git-integrated
Local IDE + Snowflake CLI	Teams with established local dbt Core workflows	Familiar tools, deploy to Snowflake via CI/CD

All three environments support the same dbt project files and produce the same results when deployed.

Documenting sources and models¶

Well-documented dbt projects are easier to onboard new engineers, debug failures, and audit for compliance. dbt provides three key documentation surfaces that integrate with Snowflake’s tooling, and CoCo can accelerate all of them by reading the Snowflake Horizon Catalog to generate accurate documentation from existing metadata.

Document your sources (sources.yml)¶

A sources.yml file declares the raw tables your dbt project depends on and provides metadata that flows directly into the Snowsight dbt project object details page. Key properties to define:

Source descriptions explaining where the data comes from and what the source system is.
Table descriptions documenting what each table contains and its business context.
Column descriptions defining the meaning of each column for downstream consumers.

CoCo can scan the Snowflake Horizon Catalog (table comments, column comments, and tags) and generate a sources.yml with accurate descriptions already populated. This saves manual discovery work and ensures your documentation stays in sync with what’s actually in your Snowflake account.

Document your models (models.yml)¶

A models.yml file (sometimes called schema.yml) describes the models your project produces. Define these properties alongside your model SQL:

Model descriptions explaining what the model does and who consumes it.
Column descriptions with business definitions and expected data types.
Meta fields for ownership, SLA expectations, domain tagging, or any team-specific conventions.

CoCo can read the Snowflake objects that your models produce and generate model documentation from the catalog metadata, keeping your models.yml files accurate as your pipeline evolves.

Maintain a project overview (overview.md)¶

The dbt project object details page in Snowsight renders an overview.md file from the project root as a full Markdown document. Use it as a living README for your data project:

Project purpose and scope.
Key models, their relationships, and data flow summary.
Team ownership and contact information.
Data refresh cadence and pipeline schedules.

When large structural changes are made to the pipeline (new domains, deprecated models, schema reorganizations), ask CoCo to regenerate or update overview.md to keep it fresh. This ensures anyone browsing the project in Snowsight gets an accurate, up-to-date picture of what the pipeline does and how it’s organized.

Concurrent execution and large projects¶

As your dbt practice matures, understanding how to structure projects for efficiency and maintainability becomes important.

Why teams start with a single dbt project object¶

Keeping your pipeline in a single dbt project object preserves end-to-end lineage across all models. This means your dbt DAG in Snowsight shows the complete dependency graph, making impact analysis and debugging straightforward. You can still run specific slices of the pipeline using --select on different task schedules while maintaining that unified view.

For example, you might schedule hourly runs for time-sensitive models while running the full pipeline daily:

-- Hourly: only the time-sensitive slice
CREATE OR ALTER TASK my_db.my_schema.hourly_slice
  WAREHOUSE = transform_wh
  SCHEDULE = '1 hour'
AS
  EXECUTE DBT PROJECT my_db.my_schema.my_project
    args='run --target prod --select my_model_a my_model_b';

-- Daily: the full pipeline
CREATE OR ALTER TASK my_db.my_schema.daily_full
  WAREHOUSE = transform_wh
  SCHEDULE = '24 hours'
AS
  EXECUTE DBT PROJECT my_db.my_schema.my_project
    args='build --target prod';

When teams outgrow a single project¶

As teams grow, they may split their pipeline into logical units for efficiency (separate domains, teams, or cadences). When this happens, maintaining dependencies across project boundaries becomes important.

Current limitation: one concurrent execution per object¶

A single dbt project object supports one concurrent execution at a time. If a second EXECUTE DBT PROJECT is issued while one is already running, it fails.

Workarounds:

Duplicate the dbt project object: Deploy multiple dbt project objects from the same source. Each object can then be executed individually with different --select arguments. For a code example, see Workaround: Duplicate the dbt project object.
Use --select to run different slices of your DAG and connect them together using the AFTER clause in tasks. Each slice runs sequentially, and you get time-appropriate scheduling.
Use dbt threads to run independent models concurrently within a single execution. This is intra-project parallelism and is fully supported.

File limit considerations¶

A dbt project object supports up to 20,000 files. For very large projects approaching this limit, consider splitting into logical sub-projects. If your project still requires more capacity, contact your Snowflake account representative.

For more details on current limitations, see Limitations, requirements, and considerations for dbt Projects on Snowflake.

Data quality testing¶

Data quality checks should be built into your dbt pipeline, not bolted on as an afterthought. dbt’s testing framework runs alongside your models and fails the pipeline when quality checks don’t pass.

Built-in dbt tests¶

Every dbt project should use the standard test types for basic data integrity:

not_null: Ensures critical columns never contain null values.
unique: Validates that primary keys and business keys are unique.
accepted_values: Confirms categorical columns contain only expected values.
relationships: Verifies referential integrity between models.

Ask CoCo to define these in your schema.yml files alongside your model definitions.

Use dbt-expectations for advanced testing¶

For teams that need expressive, verbose data quality assertions, the dbt-expectations (https://hub.getdbt.com/metaplane/dbt_expectations/latest/) package extends dbt’s testing capabilities significantly. It supports patterns like:

Row count comparisons between tables.
Distribution checks (values within expected ranges).
Pattern matching and regex validation.
Cross-column consistency checks.

dbt Projects on Snowflake has full dbt deps support, which means you can install and use any package from the dbt package hub, including dbt-expectations.

Enforce test gates in CI/CD¶

Configure your CI/CD pipeline to run dbt build (which includes both run and test) rather than dbt run alone. This ensures that:

Tests run after every model build in CI.
A failing test blocks the pull request from merging.
Data quality is validated before changes reach production.

Tests are part of dbt build, so no separate orchestration step is needed.

Third-party packages¶

dbt Projects on Snowflake has full dbt deps support, so you can install and use any package from the dbt package hub (https://hub.getdbt.com/). This includes popular packages like dbt_utils, dbt_expectations, and dbt_project_evaluator.

If you run dbt deps inside a Snowflake workspace or during dbt project object execution, Snowflake needs network access to download packages from the internet. Configure an external access integration on your dbt project object to allow this. Alternatively, run dbt deps locally or in your CI/CD pipeline and deploy the project with dbt_packages/ already included.

For details on installing and managing packages, see Understand dependencies for dbt Projects on Snowflake.

Semantic views¶

Codifying your semantic views within a dbt pipeline ensures they’re version-controlled, testable, and reproducible across environments.

Use the Snowflake Semantic Views dbt package¶

The Snowflake Semantic View dbt Package (https://hub.getdbt.com/Snowflake-Labs/dbt_semantic_view/latest/) is the recommended approach for managing semantic views in dbt rather than creating them manually through the Snowflake UI. This package lets you define semantic views as dbt models, which means:

Semantic view definitions are stored in your Git repository alongside your transformation logic.
Changes to semantic views go through the same CI/CD and review process as your models.
You get reproducibility across environments (dev, staging, prod) and a clear audit trail of who changed what and when.

Snowflake semantic views don’t support the Open Semantic Interface (OSI) and don’t integrate directly with the dbt Labs MetricFlow semantic layer. The Snowflake Semantic View dbt Package is the recommended path for codifying semantic views within your dbt pipeline on the Snowflake platform.

SQL pass-through for latest features¶

The Snowflake Semantic View dbt Package uses SQL pass-through, which means it supports the latest Snowflake SQL features regardless of the package’s version on the dbt package hub. You don’t need to wait for a package update to use new Snowflake capabilities in your semantic view definitions.

For best practices on developing and maintaining semantic views, see Best practices for semantic views.