dbt Cloud¶

Overview¶

Horizon Catalog ingests dbt model metadata and model-to-model lineage from a scheduled dbt Cloud job, through the dbt Cloud API. After the initial connection, the job keeps your metadata up to date automatically. By the time you finish this guide you’ll have:

1. A dbt Cloud service token Horizon Catalog uses to authenticate.
2. A dbt Cloud job that generates the build artifacts on a daily schedule.
3. The job ID that tells Horizon Catalog which job to read.
4. An active dbt connection in the Horizon Catalog UI.

Prerequisites¶

• A warehouse data source already connected in Horizon Catalog. The dbt connector enriches an existing data source rather than creating one.
• A dbt Cloud service token with Metadata Only and Read Only permissions. For dbt Enterprise accounts, the token needs the Job Viewer (https://docs.getdbt.com/docs/cloud/manage-access/enterprise-permissions#job-viewer) and Account Viewer (https://docs.getdbt.com/docs/cloud/manage-access/enterprise-permissions#account-viewer) permissions instead.
• Access to the Horizon Catalog connector wizard: sign in to Snowflake, open Catalog > Connections, and select dbt from the Data pipeline tools section.

1. Configure a dbt Cloud job¶

Horizon Catalog reads the build artifacts that a dbt Cloud job produces, so the job has to generate documentation, compile the project, run tests, and run on a schedule:

1. In dbt Cloud, create a job and select the environment to run it in.

2. Enable Generate docs on run. Horizon Catalog uses this step for catalog.json.

3. Add the command dbt compile --full-refresh. Horizon Catalog uses this compiled manifest for model metadata and lineage. --full-refresh compiles incremental models as a full rebuild, which keeps upstream lineage in the compiled SQL.

4. Add the command dbt test. Horizon Catalog reads test outcomes from this step, including failures. dbt compile and Generate docs on run do not execute tests, so run_results.json either omits those tests or records synthetic success. Failed tests never appear unless dbt test or dbt build actually runs the tests.

   Note

   If the job already runs dbt build, omit a separate dbt test command. dbt build runs models and tests, and Horizon Catalog still ingests the manifest and failed tests when the job ends in Error. Keep dbt compile --full-refresh in the job: that command is the preferred source for incremental-model lineage.

5. Schedule the job to run daily at 4:00 a.m., so it finishes before the Horizon Catalog ingestion runs and your metadata stays current.

6. Save the job.

7. Select Run now and wait until the run finishes. A run that ends in Error because a test failed is still usable.

   Warning

   Don’t connect the job if compile failed. A truncated manifest can remove models from Horizon Catalog.

2. Get the job ID¶

Open the job in dbt Cloud and copy the job ID from the URL. The job ID is the last number in the path:

https://<access-url>/deploy/<account-id>/projects/<project-id>/jobs/<job-id>

Here <access-url> is your dbt Cloud access URL: cloud.getdbt.com for most accounts, or an account-specific URL such as ab123.us1.dbt.com if your account has been migrated.

3. Create a dbt connection¶

In Snowflake, open Catalog > Connections and select dbt from the Data pipeline tools section. Set dbt type to dbt Cloud, then fill out the setup form with the following fields:

Select Next to continue to the Load data step, where Horizon Catalog ingests the dbt metadata. After the initial connection, the scheduled dbt Cloud job keeps your metadata up to date. No further action is required.

Troubleshooting¶

Connection security¶

All communication between Horizon Catalog and dbt Cloud uses HTTPS (TLS 1.2 or higher). Authentication uses a dbt Cloud service token, which is stored as an encrypted secret. No additional encryption configuration is required.

For more details, see Security and data protection.