Manual SPCS worker setup¶

This page is for operators who need to create Snowpark Container Services (SPCS) Worker objects by hand. For most projects, prefer the Snowflake AIM Agent for Data Warehouses or SnowConvert AI CLI (scai data worker setup), which create the secret, network rule, external access integration, and service for you. Manual creation is not recommended unless your environment requires it.

For deployment scenarios and when to use SPCS versus customer-hosted Workers, see Deploying workers.

Prerequisites¶

Prepare these objects before creating a Worker service (the agent or CLI can help with images; you usually create the compute pool yourself):

1. Compute pool — hosts the SPCS services.

CREATE COMPUTE POOL <compute_pool_name>
  MIN_NODES = 1
  MAX_NODES = <max_nodes>
  INSTANCE_FAMILY = CPU_X64_S;

INSTANCE_FAMILY selects the machine type for each node in the pool (vCPU, memory, storage, and credit consumption), similar to choosing a warehouse size. Available families vary by cloud provider and region. For the full list and sizing guidance, see Snowpark Container Services: Understanding instance families and Working with compute pools. To list families available in your account, run:

SHOW COMPUTE POOL INSTANCE FAMILIES;

Choose a family large enough for your Worker’s resource requests (for example memory and CPU in the service specification). Start with a smaller CPU family such as CPU_X64_S for trials, then size up if Workers are CPU- or memory-bound.

2. Image repository — stores Orchestrator and Worker container images. Your role needs WRITE on the repository.

3. Container images — push Orchestrator and Worker images to the repository. The Snowflake AIM Agent for Data Warehouses or SnowConvert AI CLI can help with image preparation and upload.

4. Warehouse — a warehouse for the service specification’s QUERY_WAREHOUSE.

Network access objects¶

Workers in SPCS need outbound access from Snowflake to the source database and, for some platforms, to package hosts where drivers are downloaded at container startup. Create a network rule and an external access integration, then attach the integration to the Worker service.

Create the secret, network rule, and external access integration in a database and schema your role can manage. The examples below use placeholders; they are not tied to the AIM DMV metadata database.

Setup flow¶

1. Create a secret with source database credentials (if the Worker reads them from Snowflake).
2. Create a network rule listing every host and port the Worker must reach (source database and any driver download hosts).
3. Create an external access integration that references the network rule.
4. Create or alter the Worker service with EXTERNAL_ACCESS_INTEGRATIONS = (<integration_name>).

CREATE OR REPLACE SECRET <database>.<schema>.<source_type>_SECRET
  TYPE = PASSWORD
  USERNAME = '<username>'
  PASSWORD = '<password>';

CREATE OR REPLACE NETWORK RULE <database>.<schema>.<source_type>_EGRESS_RULE
  MODE = EGRESS
  TYPE = HOST_PORT
  VALUE_LIST = (
    '<source_host>:<source_port>',
    -- driver download hosts (see per-platform lists below)
  );

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION <source_type>_EAI
  ALLOWED_NETWORK_RULES = (<database>.<schema>.<source_type>_EGRESS_RULE)
  ENABLED = TRUE;

When you use SnowConvert AI CLI (scai data worker setup) with a SQL Server source connection, the CLI can create these objects automatically. Use this page when you need to create them yourself (for example for other platforms, or when you use --skip-egress).

Allowing SPCS to reach your source system (inbound access)¶

An egress network rule only controls what the Worker is allowed to call out to. Your source system also needs to accept the connection: SPCS containers connect from Snowflake-managed IP addresses, not from an address you control, so the source system’s firewall, security group, or VPC ingress rules must explicitly allow those addresses.

1. Get the current Snowflake egress IP ranges for your account and region:

SELECT SYSTEM$GET_SNOWFLAKE_EGRESS_IP_RANGES();

2. Add those ranges to the source system’s inbound allowlist (for example a security group rule for the source’s port, a firewall rule, or a VPC peering/PrivateLink configuration, depending on how the source is hosted).
3. Re-run the function periodically or before major changes: Snowflake can update egress IP ranges, and a stale allowlist causes connection failures on the Worker side even though the SPCS-side network rule and external access integration are configured correctly.

Worked example: Redshift Worker service¶

The following sketch assembles a Worker service for a Redshift source using UNLOAD extraction. Replace every placeholder with your own values; this isn’t a copy-paste-ready script.

-- Secret and network access
CREATE OR REPLACE SECRET <database>.<schema>.REDSHIFT_SECRET
  TYPE = PASSWORD
  USERNAME = '<username>'
  PASSWORD = '<password>';

CREATE OR REPLACE NETWORK RULE <database>.<schema>.REDSHIFT_EGRESS_RULE
  MODE = EGRESS
  TYPE = HOST_PORT
  VALUE_LIST = (
    '<redshift_endpoint>:5439'
    -- plus driver download hosts, see "Driver download hosts by platform" below
  );

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION REDSHIFT_EAI
  ALLOWED_NETWORK_RULES = (<database>.<schema>.REDSHIFT_EGRESS_RULE)
  ENABLED = TRUE;

-- Worker service
CREATE SERVICE <worker_service_name>
  IN COMPUTE POOL <compute_pool_name>
  FROM SPECIFICATION $$
    spec:
      containers:
        - name: agent
          image: <your_image_repository>/data-exchange-agent:redshift
          env:
            # Source system
            DATA_SOURCE_HOST: <redshift_endpoint>
            DATA_SOURCE_PORT: 5439
            DATA_SOURCE_DATABASE: <source_database>

            # Snowflake
            SNOWFLAKE_WAREHOUSE: <warehouse_name>

            # Worker tuning
            MAX_PARALLEL_TASKS: 3
            AGENT_AFFINITY: <affinity_tag>

            # Required only when using the "unload" extraction strategy
            UNLOAD_S3_BUCKET: <bucket_name>
            UNLOAD_IAM_ROLE_ARN: <iam_role_arn>
          resources:
            requests:
              memory: 10G
              cpu: 6
            limits:
              memory: 10G
              cpu: 6
          secrets:
          - snowflakeSecret: REDSHIFT_SECRET
            secretKeyRef: USERNAME
            envVarName: DATA_SOURCE_USERNAME
          - snowflakeSecret: REDSHIFT_SECRET
            secretKeyRef: PASSWORD
            envVarName: DATA_SOURCE_PASSWORD
  $$
  EXTERNAL_ACCESS_INTEGRATIONS = (REDSHIFT_EAI)
  MIN_INSTANCES = 1
  MAX_INSTANCES = 1;

UNLOAD_S3_BUCKET and UNLOAD_IAM_ROLE_ARN are only needed when a table’s extraction.strategy is unload; see Migrating Data from Amazon Redshift and Extraction strategies. When you use affinity routing, set AGENT_AFFINITY to match the workflow’s affinity value (or a matching wildcard). See Affinity.

Also complete the inbound access step above: your Redshift cluster’s security group must allow connections from your account’s Snowflake egress IP ranges on port 5439.

Driver download hosts by platform¶

Workers download some database drivers when the container starts. Include your source host and port in every network rule, plus the driver download hosts for your platform.

SQL Server and Azure Synapse¶

Authoritative allowlist (plus your source host):

• packages.microsoft.com:443
• pmc-geofence.trafficmanager.net:443
• deb.debian.org:80
• deb.debian.org:443

The Worker installs the Microsoft ODBC Driver for SQL Server from Microsoft’s apt repository at startup.

Amazon Redshift¶

Recommended allowlist (plus your Redshift endpoint):

• s3.amazonaws.com:443 (Redshift ODBC driver .deb download)
• deb.debian.org:80
• deb.debian.org:443

PostgreSQL¶

Recommended allowlist (plus your PostgreSQL endpoint):

• deb.debian.org:80
• deb.debian.org:443

The Worker installs odbc-postgresql from Debian package mirrors at startup.

Oracle¶

No driver download hosts. The Worker uses the pre-installed thin oracledb driver. Allow only your Oracle listener host and port.

Teradata¶

No driver download hosts. The Worker uses the pre-installed teradatasql driver. Allow only your Teradata host and port.

Related content¶

• Deploying workers
• Data migration (CLI)
• Data migration configuration reference
• The SNOWCONVERT_AI database