Redshift provisioned cluster¶

This guide walks through connecting a Redshift provisioned cluster to Horizon Catalog. The connector uses a customer-owned IAM role to discover the cluster endpoint, read audit logs, and query Redshift system views for metadata and lineage.

Prerequisites¶

• Access to an AWS user with permission to modify IAM, Redshift parameter groups, and the cluster.
• Access to the Redshift cluster as an admin user.
• Open the Horizon Catalog connector wizard in another browser tab: sign in to Snowflake, go to Catalog > Connections, and select Redshift. You’ll need the IAM Principal and External ID values shown in the Authorize AWS access step when you create the IAM role in Step 5.

1. Create a new data source¶

In Snowflake, open Catalog > Connections and select Redshift from the list of connectors. You will go through a 4-step wizard: Select a connection → Add data source → Authorize AWS access → Select databases.

In the Add data source step, fill in the following fields:

2. Enable audit logging¶

Horizon Catalog reads connection and user activity logs from your cluster’s audit logs. Enable audit logging on the cluster and choose where Redshift delivers the logs. Horizon Catalog auto-detects the destination, so you only need to configure one.

1. In the Redshift console, go to Clusters and select your cluster.
2. Open the Properties tab, scroll to Database configurations, select Edit, then choose Edit audit logging from the dropdown menu.
3. Turn on Configure audit logging, then choose a log export type:
   • CloudWatch: Redshift delivers audit logs directly to Amazon CloudWatch Logs, with no S3 bucket to manage. AWS recommends this option.
   • S3 bucket: Redshift writes audit logs to an S3 bucket you specify. Select an existing bucket or create a new one.
4. Select Save changes.

For details, see Database audit logging (https://docs.aws.amazon.com/redshift/latest/mgmt/db-auditing.html) in the AWS documentation.

3. Enable user activity logging¶

The user activity log is where Redshift records the SQL statements that Horizon Catalog uses to build popularity and lineage. Enabling audit logging in Step 2 captures the connection log and user log, but the user activity log is produced only when you also set the enable_user_activity_logging parameter to true in the cluster’s parameter group.

1. In the Redshift console, go to Parameter groups and create a custom group if needed:

   • Base it on the same family as the existing group.
   • Apply the new group to your cluster under Clusters > Modify > Parameter group.

2. Open the custom parameter group and set:

   Parameter	Value
   enable_user_activity_logging	true

   Expand

   Show lessSee more

3. Reboot the cluster if the parameter group change requires it.

For details, see Managing parameter groups (https://docs.aws.amazon.com/redshift/latest/mgmt/managing-parameter-groups-console.html) in the AWS documentation.

4. Create the Horizon Catalog database user¶

Connect to each database you want to ingest and run:

CREATE USER horizoncatalog WITH PASSWORD DISABLE syslog ACCESS UNRESTRICTED;
GRANT SELECT ON SVV_TABLE_INFO TO horizoncatalog;
GRANT SELECT ON SVV_TABLES TO horizoncatalog;
GRANT SELECT ON SVV_COLUMNS TO horizoncatalog;
GRANT SELECT ON STL_QUERYTEXT TO horizoncatalog;
GRANT SELECT ON STL_DDLTEXT TO horizoncatalog;
GRANT SELECT ON STL_QUERY TO horizoncatalog;

Repeat in every database you select in the wizard. The grants are database-scoped — running them in one database does not apply to others.

5. Create an IAM role¶

Horizon Catalog uses a customer-owned IAM role that it assumes to access the cluster and its logs. In the AWS IAM console, go to Roles and select Create role, then add a custom trust policy and an inline permission policy as follows.

5a. Trust policy¶

In the Horizon Catalog wizard, go to the Authorize AWS access step. Horizon Catalog pre-generates two values you must copy into the trust policy:

• External ID — a unique token generated per data source
• IAM Principal — the Horizon Catalog AWS principal that will assume your role

For Trusted entity type, select Custom trust policy and paste the following, substituting the IAM Principal and External ID values from the wizard:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<IAM Principal from wizard>" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": { "sts:ExternalId": "<External ID from wizard>" }
      }
    }
  ]
}

5b. Permission policy¶

Select Next. On the Add permissions step, select Create inline policy, then open the JSON editor and paste the following. Replace {region_id}, {account_id}, {cluster_name}, and {bucket} with your values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RedshiftClusterAccess",
      "Effect": "Allow",
      "Action": [
        "redshift:DescribeClusters",
        "redshift:DescribeLoggingStatus",
        "redshift:GetClusterCredentials"
      ],
      "Resource": [
        "arn:aws:redshift:{region_id}:{account_id}:cluster:{cluster_name}",
        "arn:aws:redshift:{region_id}:{account_id}:dbuser:{cluster_name}/horizoncatalog",
        "arn:aws:redshift:{region_id}:{account_id}:dbname:{cluster_name}/*"
      ]
    },
    {
      "Sid": "S3AuditLogAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetBucketLogging",
        "s3:GetBucketAcl",
        "s3:GetBucketPolicy",
        "s3:GetBucketVersioning",
        "s3:GetBucketPublicAccessBlock",
        "s3:GetBucketTagging",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionTagging",
        "s3:ListBucketVersions",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:GetInventoryConfiguration",
        "s3:GetLifecycleConfiguration",
        "s3:GetBucketOwnershipControls",
        "s3:GetBucketPolicyStatus"
      ],
      "Resource": ["arn:aws:s3:::{bucket}", "arn:aws:s3:::{bucket}/*"]
    },
    {
      "Sid": "CloudWatchLogsAccess",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:StartQuery",
        "logs:GetQueryResults"
      ],
      "Resource": "*"
    }
  ]
}

Enter a name and description for the role, then select Create role. For example:

• Role name: SnowflakeHorizon-Redshift
• Description: IAM role used to connect Horizon Catalog to Amazon Redshift

Open the role’s summary page and copy the role ARN — you paste it into the wizard in Step 6.

6. Confirm authorization and choose databases¶

Return to the Authorize AWS access step in the Horizon Catalog wizard. The External ID and IAM Principal fields are already pre-filled. Paste the Role ARN from Step 5 into the Role ARN field and select Next.

In the Select databases step, choose the databases and schemas you want Horizon Catalog to crawl, then select Done.

Initial metadata typically appears within a few minutes. Popularity and lineage can take 24–48 hours as query log history accumulates.

Troubleshooting¶