Tutorial: Set up CI/CD integrations on dbt Projects on Snowflake¶
Introduction¶
This tutorial is a continuation of the Getting started with dbt Projects on Snowflake tutorial. It assumes you’ve already completed that tutorial and have a working Snowflake environment with your database, schemas, warehouse, and source data set up.
This tutorial guides you through building a secure CI/CD pipeline for dbt Projects on Snowflake using GitHub Actions, OIDC authentication, Snowflake CLI, and dbt project objects to automate testing, deployment, and orchestration with minimal overhead.
For more information, see CI/CD integrations on dbt Projects on Snowflake.
Overview¶
This tutorial walks you through the following steps:
-
Setting up your Snowflake environment:
- You choose one of three ways to prepare dev and prod targets (full database clone, partial clone, or brand-new databases).
- Your dbt project must include a
profiles.ymlthat refers to these dev and prod targets.
-
Setting up an OIDC service user for secure authentication: Instead of passwords or long-lived tokens, you create a Snowflake service user that trusts GitHub through OpenID Connect. This enables secure, short-lived, per-run authentication.
-
Setting up network policies: (Optional) If your Snowflake account restricts inbound IPs, you can use Snowflake-managed network rules to add Github Actions runner IPs to your service user’s network policy. Otherwise, you can skip this step.
-
Storing GitHub secrets and repository variables to configure Snowflake CLI in your workflows:
- Your Snowflake account identifier
- (Optionally) the Snowflake username
- The target database and schema where dbt project objects will be deployed
-
Creating GitHub Actions workflows:
- CI workflow that triggers on pull requests, deploys a tester dbt project object, and runs
dbt buildto build models and test them in DAG order. If anything breaks, the pull request fails.- CD workflow that triggers on merges to main, deploys the production dbt project object, and optionally applies scheduling.
At the end of the tutorial, you will have:
- A fully automated, GitHub-driven dbt workflow
- Secure OIDC authentication
- Consistent, tested deployments into Snowflake
- Version-controlled orchestration (optional)
- A repeatable template for scaling dbt workflows across teams
Prerequisites¶
-
GitHub
- An existing dbt Project in a GitHub account that can create a repository and manage access to that repository.
-
Snowflake
- Completion of the
Getting started with dbt Projects on Snowflake tutorial,
which sets up the
tasty_bytes_dbt_dbdatabase,dev/prodschemas,tasty_bytes_dbt_whwarehouse, and source data. - Basic understanding of dbt Projects on Snowflake. For more information, see dbt Projects on Snowflake.
- A Snowflake account and user with privileges as described in Access control for dbt projects on Snowflake.
- Privileges or administrator assistance to create and edit the following:
- GitHub repository secrets to specify the account and (optional) username
- A Snowflake service user
- Network policy
- Completion of the
Getting started with dbt Projects on Snowflake tutorial,
which sets up the
Set up your environment¶
Set up where your dbt project will read and write in Snowflake, then update your profiles.yml file.
Create dev and prod databases and schemas¶
Note
If you’ve already completed the
Getting started with dbt Projects on Snowflake tutorial
and run the tasty_bytes_setup.sql file, your database (tasty_bytes_dbt_db) and schemas (dev, prod) already exist. You can
skip this step. For details, see
Run the SQL commands in tasty_bytes_setup.sql to set up source data.
To set up where your dbt project will read and write in Snowflake, choose one of the following options:
- Create an empty database with dev and prod schemas
- Clone your production database using zero-copy cloning
- Create an empty dev database and clone the production schemas you need
Create an empty database with dev and prod schemas¶
This is the simplest approach when you’re starting from scratch.
Clone your production database¶
Use Snowflake’s zero-copy cloning to create a full replica of your production database, as shown in the following example. This gives you a high-fidelity testing environment and is cost-effective because you only pay storage for tables that change during dbt runs.
Create an empty dev database and clone the production schemas you need¶
Use this method when you only need specific schemas for testing.
Update your profiles.yml file¶
To manage CI/CD for a dbt project object in GitHub Actions, you must include a profiles.yml file inside your dbt project folder (for
example, my_dbt_project/profiles.yml). This file defines your dev and prod targets and uses placeholder values that GitHub repository secrets will later replace.
Edit this file directly on GitHub to reference the dev and prod databases and schemas you created, as shown below:
Key points from the example:
target: devsets the default target of the dbt project. This value can be overridden by Snowflake CLI or a dbt project object.devandprodboth usetype: snowflake.- Database and schema point to the databases and schemas you created in the previous step (or already set up from the getting-started tutorial).
- Warehouse is the warehouse created in the getting-started tutorial (
tasty_bytes_dbt_wh). - Account and user are set to dummy values like ‘_’ because they’ll be replaced by GitHub repository secrets later.
threads: 8sets the number of concurrent threads dbt uses. Snowflake recommends 8 threads.
Create a GitHub service user in Snowflake (recommended)¶
GitHub Actions run using the Snowflake user specified in your Snowflake CLI commands. To keep things clean and secure, create a dedicated Snowflake user for all GitHub workflows and grant it the required privileges.
Recommended: OIDC-based service user¶
This approach uses OpenID Connect (OIDC) rather than long-lived credentials. The service user trusts GitHub as an identity provider, allowing
GitHub Actions to request short-lived tokens for each workflow run. You will map this user to an environment subject like
environment:prod in a later step.
Each OIDC service user must have a unique subject. We recommend using a repo path and an environment name, for example
repo:<org>/<repo>:environment:<environment_name>. The environment name can be anything, as long as it matches exactly in your GitHub
Action YAML file. For more information, see Workload identity federation.
Create an OIDC-based service user as follows:
After you create your user, explicitly grant the default role for the service user to assume that role. The DEFAULT_ROLE parameter only sets the user’s default role and doesn’t grant it.
Set a default warehouse:
Alternative: PAT-based authentication (less secure)¶
If you prefer to use one Snowflake user across multiple repositories, or cannot use OIDC, you can create the user with a personal access token (PAT) instead.
This method is easier to reuse across repositories but less secure because it relies on long-lived credentials and requires manual rotation.
(Optional) Set up a network policy for GitHub Actions¶
Now that you’ve created the service user that Snowflake CLI will use, let’s configure this user to connect to your Snowflake account from within GitHub Actions.
Note
Creating or modifying network policies requires ACCOUNTADMIN or an equivalent role.
Determine whether you need a network policy¶
- If your account restricts inbound access, you must create or update a network policy to add GitHub Actions runner IPs to your allowlist. Snowflake simplifies this with Snowflake-managed network rules. For more information, see Network rules.
- If your account does not restrict inbound access, no network policy changes are required.
If you’re unsure, skip this step for now and return only if you see an error like: Incoming request with IP/Token <IP> is not allowed to access Snowflake.
To create and apply a network policy to a user, choose one of the following options:
- Create a new network policy and assign it to the service user, or
- Add the GitHub Actions network rule to an existing network policy that the user already uses.
Note
Before doing this, consult your Snowflake account admin. They must ensure the policy includes not only the GitHub Actions network rule but also any other IP ranges your organization requires.
Once a network policy is applied, Snowflake restricts user access based on its allowed and blocked IP ranges. Your account admin might need to adjust the policy or apply it account wide to avoid unintentionally blocking essential access.
Option 1: Create a new network policy and apply it to the user¶
A Snowflake user can have only one network policy at a time. If the user doesn’t have one or you want to replace the existing policy, complete the following steps:
Option 2: Add a network rule to an existing network policy¶
If the user already has a network policy, you can add the GitHub Actions rule to it.
Note
If the network policy is applied at the account level or shared by many users, updating it will affect everyone.
The user inherits the update automatically since they’re already assigned to this policy.
Configure GitHub repository secrets and variables¶
GitHub Actions use the Snowflake CLI to connect to your Snowflake account, so you must configure GitHub repository secrets and variables first. This is how the CI/CD integration passes Snowflake account info into Snowflake CLI inside GitHub Action workflows.
Configure GitHub repository secrets¶
Add secrets to securely store the information Snowflake CLI needs to identify your Snowflake account and, if required, the user it should authenticate as:
-
In your GitHub repository, go to Settings.
-
From the left-hand side navigation, select Secrets and variables » Actions.
-
Under Secrets, select New repository secret.
-
Add a secret to connect your Snowflake account:
- Name:
SNOWFLAKE_ACCOUNT - Value: Your Snowflake account identifier (for example,
org_name-account_name). This value tells Snowflake CLI which account you want to connect to.
- Name:
-
Select Add secret.
-
(Optional) If you aren’t using OIDC, select New repository secret to specify the Snowflake username the CLI should use when connecting. It specifies which user credentials to run commands under.
- Name:
SNOWFLAKE_USER - Value: Optional if you’re using OIDC or credential-less authentication.
- With OIDC, Snowflake CLI automatically matches the GitHub Action’s subject to the OIDC service user (created in Step 3), so this is not required.
- Without OIDC, you must specify a user (and supply password or key credentials). As a recommended best practice, you should create a personal access token in Snowsight. For more information, see Generating a programmatic access token.
- Name:
-
Select Add secret.
-
(Optional) If you aren’t using OIDC, select New repository secret to specify the service user’s personal access token that the CLI should use when connecting.
- Name:
SNOWFLAKE_PAT - Value: Optional if you’re using OIDC or credential-less authentication.
- Name:
Configure GitHub repository variables¶
These help Snowflake CLI connect to the right database and schema. Complete the following steps:
-
In your GitHub repository, go to Settings.
-
From the left-hand side navigation, select Secrets and variables » Actions.
-
Under Variables, select New repository variable.
-
Add a database variable:
- Name:
SNOWFLAKE_DATABASE - Value: Enter an existing database where the dbt project object will be created.
- Name:
-
Select Add variable.
-
Add a schema variable:
- Name:
SNOWFLAKE_SCHEMA - Value: Enter an existing schema where the dbt project object will be created.
- Name:
-
Select Add variable.
Create your Continuous Integration (CI) GitHub Action¶
This step is where automation starts. This CI workflow runs whenever a pull request targets main. It:
- Creates a tester dbt project object in Snowflake
- Runs
dbt buildagainst your dev target, which builds all models and runs tests in DAG order, failing early if any test fails - Fails the pull request if the dbt execution fails
Create your CI workflow file¶
-
In your GitHub repository, go to Actions.
-
From the left-hand side navigation, select New workflow.
-
Select set up a workflow yourself to create an empty workflow.
-
Name the file
incoming_pr.yml. -
Copy and paste the following into the file:
-
Select Commit changes.
-
Select Create a new branch for this commit and start a pull request.
-
Select Propose changes.
-
After you finish submitting the pull request, you should see your
incoming_pr.ymlaction start to run. -
After it’s merged, the file will be saved to
.github/workflows/incoming_pr.yml.
Key pieces from the workflow file¶
-
Triggers on pull requests to
main: -
Grants permissions and sets environment variables for Snowflake CLI:
-
Steps in the job:
-
Check out repository code (
actions/checkout@v4). -
Install Snowflake CLI using
snowflakedb/snowflake-cli-action@v2.0withuse-oidc: true. -
Run
snow --version. -
Run
snow connection test -xto verify OIDC connection. -
Deploy a tester dbt project object using
snow dbt deploy ... --force -x(with--sourceif the dbt project is in a subfolder). -
Run
snow dbt list -xto show dbt project objects. -
Build and test the dbt project in DAG order:
snow dbt execute -x tester_tasty_bytes_dbt_project_object_gh_action build --target devUsing
buildinstead of separaterunandtestcommands ensures that tests execute immediately after each model is built, in dependency order. If an upstream model’s test fails, downstream models aren’t built, providing faster feedback and preventing invalid data from propagating.
-
-
Once you commit this new workflow on a branch and open a pull request, GitHub Actions will run it. If the dbt project object fails to build a model or any test fails, the CI check fails and the pull request can’t be merged.
Create your Continuous Deployment (CD) GitHub Action¶
The CD workflow runs after code is merged to main (or any direct push to main), ensuring the dbt project object in Snowflake reflects the latest code.
Create your CD workflow file¶
-
In your GitHub repository, go to Actions.
-
From the left-hand side navigation, select New workflow.
-
Select set up a workflow yourself to create an empty workflow.
-
Name the file
pr_merged.yml. -
Copy and paste the following into the file:
-
Select Commit changes to save the file to
.github/workflows/pr_merged.yml. -
Navigate to the Actions tab of your repository to see your
pr_merged.ymlaction start to run.
Key pieces from the workflow file¶
-
Triggers on pushes to
main: -
Similar permissions/env as CI:
-
Steps in the job:
- Check out the repository code.
- Install Snowflake CLI with OIDC.
- Run
snow --version. - Run
snow connection test -xto verify OIDC connection. - Deploy/update the production dbt project object with
snow dbt deploy ... --default-target prod --force -x. - Run
snow dbt list -xto show dbt project objects. - (Optional) Run a
schedules.sqlfile to manage tasks (see next section).
-
Once this workflow is in place, every successful merge to main (or push) updates the dbt project object in Snowflake.
(Optional) Add orchestration with Snowflake tasks¶
Orchestrate executions of your dbt project object using a schedules.sql file and Snowflake tasks (triggered from the CD workflow):
-
In your GitHub repository, navigate to your dbt project (for example,
tasty_bytes/). -
Create a file named
schedules.sqland copy and paste the following into the file.This file:
-
Suspends any existing tasks
-
Creates or alters tasks to:
- Run a subset of the DAG on a schedule, failing early if any test fails
- Run the full project, failing early if any test fails
-
Resumes tasks in the correct order (child → root)
-
-
Select Commit changes.
-
In your GitHub repository, navigate to
.github/workflows/pr_merged.ymland uncomment theRun schedules.sql to create...step at the end of the file. -
Select Commit changes.
Next steps¶
Next steps to improve your workflow:
-
Use zero-copy cloning in CI:
Test against fresh production data by adding a step in
incoming_pr.ymlbefore deploying your tester dbt object: -
Add alerting:
Configure Slack or email notifications in GitHub Actions, or use Snowflake task error notifications.
For more information, see Configure a task to send error notifications.
-
Explore Managing dbt Projects on Snowflake using Snowflake CLI.