dbt Core (open source)

Overview

Horizon Catalog ingests dbt model metadata and model-to-model lineage from the build artifacts that dbt Core produces. You generate the artifacts locally or in your continuous integration (CI) pipeline and upload them through a dbt connection, which enriches an existing warehouse data source with lineage, and with model and column descriptions for non-Snowflake warehouses. By the time you finish this guide you’ll have:

  1. The build artifacts generated from your dbt project.
  2. An active dbt connection in the Horizon Catalog UI, with the artifacts uploaded.
  3. A routine for refreshing the artifacts when your dbt project changes.

Prerequisites

  • A warehouse data source already connected in Horizon Catalog. The dbt connector enriches an existing data source rather than creating one.
  • A dbt project you can build, so that it produces the following artifacts:
    • manifest.json (required)
    • catalog.json (required)
    • run_results.json (optional, used to ingest dbt tests)
  • Access to the Horizon Catalog connector wizard: sign in to Snowflake, open Catalog > Connections, and select dbt from the Data pipeline tools section.

1. Generate the build artifacts

Run the following commands in your dbt project and collect the artifacts from the target directory.

Generate manifest.json:

dbt compile --full-refresh

Warning

dbt docs generate also writes a manifest.json, but that file doesn’t contain enough lineage information. Use the manifest.json produced by dbt compile --full-refresh, and copy it to another location before running dbt docs generate so it isn’t overwritten.

Generate catalog.json:

dbt docs generate

(Optional) Generate run_results.json for dbt tests:

dbt test

2. Create a dbt connection

In Snowflake, open Catalog > Connections and select dbt from the Data pipeline tools section. Set dbt type to dbt Core (open source), then fill out the setup form with the following fields:

FieldValue
Display NameA name for this data source, which defaults to dbt
dbt typeSelect dbt Core (open source)
Connected WarehouseThe existing warehouse data source that your dbt models are built in (for example, Snowflake, BigQuery, or Amazon Redshift)
manifest.jsonDrag and drop the manifest.json file from Step 1
catalog.jsonDrag and drop the catalog.json file from Step 1
run_results.json(Optional) Drag and drop the run_results.json file. Required only if you want to ingest dbt tests
Snowflake DatabaseThe Snowflake database where Horizon Catalog stages the uploaded artifacts, which defaults to CONNECTORS
Snowflake SchemaThe Snowflake schema where Horizon Catalog stages the uploaded artifacts, which defaults to METADATA

Note

Each artifact file can be up to 250 MB.

Select Next to continue to the Load data step, where Horizon Catalog ingests the dbt metadata.

3. Update the build artifacts

Unlike dbt Cloud, dbt Core doesn’t refresh automatically. To reflect changes in your dbt project, regenerate the build artifacts (see Step 1) and upload them again:

  1. In Snowflake, open Catalog > Connections, find your dbt data source, and select Configure.
  2. Upload the updated manifest.json, catalog.json, and (optionally) run_results.json files.
  3. Complete the wizard to apply your changes.

Troubleshooting

SymptomMost likely causeWhere to look
Lineage is missing or incompleteThe uploaded manifest.json came from dbt docs generate, which doesn’t contain enough lineage informationStep 1
The file upload failsOne of the artifacts is larger than the 250 MB limitStep 2
dbt tests don’t appearrun_results.json wasn’t uploadedStep 1
Metadata stops reflecting changes to your dbt projectdbt Core artifacts don’t refresh on their own and have to be uploaded againStep 3
Lineage appears, but model and column descriptions don’tThe connected warehouse is Snowflake, which uses its own native object descriptions insteadHow the connector works

Connection security

Horizon Catalog doesn’t connect to dbt for a dbt Core connection, so no dbt credentials are involved. The build artifacts you upload are staged in your own Snowflake account, in the database and schema you select when you create the connection.

For more details, see Security and data protection.