Deploying workers¶

AIM DMV Workers connect to your source system and to Snowflake to move and compare data. This page covers the deployment scenarios Workers support and what you need for the most common option, Snowpark Container Services (SPCS).

For shared architecture and deployment options at a glance, see Data Migration & Validation overview.

Prefer automated setup¶

Creating Worker-related Snowflake objects by hand (secrets, network rules, external access integrations, and the service itself) is not recommended for routine projects. Normally the Snowflake AIM Agent for Data Warehouses or SnowConvert AI CLI (scai data worker setup) creates those objects for you after you provide connection details and a compute pool.

Use Manual SPCS worker setup only when your environment requires hand-written SQL, or when you need the per-platform driver download host lists and a full service specification example.

Deployment scenarios¶

Workers (and the Orchestrator) can run in several places. Mix and match per component: for example, run the Orchestrator on SPCS while Workers run in your data center, or the reverse.

Whichever option you choose, every Worker needs the same three things: a path to the source system, a path to Snowflake, and a Worker configuration file (TOML) describing both. See Data migration configuration reference: Worker configuration for the file format.

Keeping source credentials out of Worker TOML¶

Worker configuration examples show source credentials inline for readability. How you avoid that in practice depends on where the Worker runs:

• On SPCS, source credentials come from Snowflake SECRET objects bound into the service specification. The agent or CLI creates them for you. See Network access objects.
• On your own hosts or in Kubernetes, the Worker can resolve credentials at startup from an external secret store, referencing them in the connection fields instead of storing a password. See Resolving credentials from an external secret manager.

Snowpark Container Services¶

When to use SPCS¶

Deploy on SPCS when you want Orchestrator and Worker compute inside Snowflake rather than on customer-hosted machines. You can run both components on SPCS, both outside Snowflake, or mix the two (for example Orchestrator on SPCS and Workers in your data center).

For Workers on SPCS, the Worker needs a path to the source database, and often to public hosts where drivers are downloaded. Two paths reach the source:

• Direct egress. Snowflake opens the connection to the source host. Your account needs a network rule and an external access integration, and the source must accept connections from Snowflake egress IPs. See Allowing SPCS to reach your source system.
• Data Connectivity Proxy. Use this when the source is only reachable from a private network that Snowflake can’t egress to, and you still want the Worker on SPCS. scai data worker setup --dcp attaches that Worker to a proxy. An agent on your network opens an outbound TLS connection to Snowflake on port 443, and the Worker’s connection to the source goes through that tunnel. You don’t open an inbound port, and you don’t allowlist Snowflake egress IPs for the source host. See Reach a private source with Data Connectivity Proxy.

Driver download hosts stay on the direct egress path either way. Customer-hosted VMs or Kubernetes next to the source remain a valid choice. You can keep the Orchestrator on SPCS with either Worker placement.

What you typically prepare¶

You usually create a compute pool and warehouse before the agent or CLI deploys services. For INSTANCE_FAMILY sizing and SHOW COMPUTE POOL INSTANCE FAMILIES, see Manual SPCS worker setup.

The agent or CLI then typically creates the Worker secret, network rule, external access integration, and service. With --dcp, it also creates the Data Connectivity Proxy rule, integration, and proxy. Avoid recreating those by hand unless you have a reason to. Full SQL templates live on the manual SPCS worker setup page.

SQL Server Windows authentication is the exception. Create the Kerberos keytab and krb5.conf secrets yourself before scai data worker setup, and add each domain controller to the egress rule. See Windows authentication.

How many Worker instances¶

More Worker instances mean more concurrent extraction and load, which increases throughput. The source system is usually the constraint — keep adding Workers until you see diminishing returns or the source starts to struggle, then scale back.

Each instance runs up to MAX_PARALLEL_TASKS tasks concurrently (default 2 on the SPCS Worker image, and 4 for max_parallel_tasks in Worker TOML). Total concurrency against your source is roughly the instance count multiplied by that value, so tune both to what the source can absorb. When the source is struggling, prefer rate limiting to keep Workers up while holding back specific extraction tasks.

Prefer more Workers with a lower MAX_PARALLEL_TASKS (5 or fewer) over fewer Workers with many threads each. Size each instance at roughly 1–2 vCPU and 1–2 GB per MAX_PARALLEL_TASKS thread. scai data worker setup doesn’t apply this sizing: it creates one instance that requests cpu: 0.1 and memory: 2Gi, and it doesn’t set MAX_PARALLEL_TASKS, so the Worker runs two tasks at a time. See Instance count and resource sizing.

Orchestrator versus Worker services¶

Run both as Services (not Jobs). Both suspend themselves after a period without work. See Controlling cost on SPCS.

The Orchestrator creates the AIM DMV metadata database and schema objects on startup if they do not already exist. See The SNOWCONVERT_AI database for what that database contains after setup.

Network access for Workers¶

Workers in SPCS run inside Snowflake’s network. The Worker can’t extract or validate data without a path to the source. On the direct-egress path, Snowflake calls the source host itself:

1. Allow outbound access with a network rule and external access integration (created for you by the agent or CLI in normal setups).
2. Attach that integration to the Worker service.
3. Allow inbound connections on the source system from Snowflake egress IPs (your responsibility; see the next section).

A Worker created with --dcp doesn’t use that inbound allowlist for the source host. The source host is reached through the Data Connectivity Proxy tunnel. Driver download hosts still use the direct egress rule. See Reach a private source with Data Connectivity Proxy.

Driver download hosts and hand-written SQL for secrets, rules, and services are documented under Manual SPCS worker setup.

Allowing SPCS to reach your source system (inbound access)¶

This section is the direct-egress path. It applies when the Worker calls the source host from Snowflake. A Worker created with --dcp doesn’t use it for the source host, so you don’t allowlist Snowflake egress IPs for that host. Allowlisting SYSTEM$GET_SNOWFLAKE_EGRESS_IP_RANGES() remains the way a direct-egress source accepts the Worker.

An egress network rule only controls what the Worker is allowed to call out to. Your source system also needs to accept the connection: SPCS containers connect from Snowflake-managed IP addresses, not from an address you control, so the source system’s firewall, security group, or VPC ingress rules must explicitly allow those addresses.

1. Get the current Snowflake egress IP ranges for your account and region:

SELECT SYSTEM$GET_SNOWFLAKE_EGRESS_IP_RANGES();

2. Add those ranges to the source system’s inbound allowlist (for example a security group rule for the source’s port, a firewall rule, or a VPC peering/PrivateLink configuration, depending on how the source is hosted).
3. Re-run the function periodically or before major changes: Snowflake can update egress IP ranges, and a stale allowlist causes connection failures on the Worker side even though the SPCS-side network rule and external access integration are configured correctly.

Reach a private source with Data Connectivity Proxy¶

Use this path when the source is only reachable from a private network and you still want the Worker on SPCS. scai data worker setup --dcp <PROXY_NAME> attaches the Worker to a Data Connectivity Proxy. An agent on your network opens an outbound TLS connection to Snowflake on port 443. The Worker’s connection to the source goes through that tunnel. You don’t open an inbound port on the source, and you don’t add Snowflake egress IPs to the source host’s allowlist.

A setup that omits --dcp is unchanged. It creates the direct egress rule, puts the source host in that rule, and attaches only the direct external access integration.

scai data worker setup --compute-pool <COMPUTE_POOL> --source-connection <SOURCE_CONNECTION> --dcp <PROXY_NAME>

<PROXY_NAME> is an unquoted Snowflake identifier: a letter or underscore, then letters, digits, underscores, or $. The CLI stores it in uppercase. A quoted or invalid name fails before any object is created. --dcp and --skip-egress together also fail before any object is created. --skip-egress still means you already created the secret, rules, and integrations yourself.

The source connection’s host must be a DNS name. An IPv4 or IPv6 address is rejected before any object is created. If the source has no DNS name, add a hosts entry on the agent host and use that name in the source connection. The agent resolves the name on its own network.

--dcp applies only when the Worker service is created:

• If the service doesn’t exist, setup creates the tunnel objects and attaches them.
• If the service exists and you omit --yes, setup doesn’t attach the tunnel. It warns that --dcp applies only when the Worker is created, and tells you to run the command again with --yes.
• --yes on an existing service drops and recreates the service from the CLI specification, including the tunnel integrations. Integrations and the network rule that already match are left in place. Their created_on doesn’t change. The CLI alters those objects. It doesn’t replace them, because replacing an integration detaches it from the proxy and from services.

Several Workers can share one proxy. Each Worker has its own Data Connectivity Proxy integration, because each integration allows one source endpoint. Setup adds the new integration to the proxy and keeps every integration already listed. It won’t rewrite an integration name it can’t reproduce exactly, and it stops rather than drop another Worker’s integration.

For the object names, see Data Connectivity Proxy objects.

Privileges¶

The proxy owner mints the token and gives it to the person who starts the agent. <days> is 1 to 90. Grants that let a migration role use the proxy don’t include this call. Replacement tokens also come from the owner.

SELECT SYSTEM$GENERATE_DATA_CONNECTIVITY_PROXY_BOOTSTRAP_TOKEN('<PROXY_NAME>', <days>);

See CREATE DATA CONNECTIVITY PROXY and SYSTEM$GENERATE_DATA_CONNECTIVITY_PROXY_BOOTSTRAP_TOKEN.

Creating a proxy runs SELECT SYSTEM$ISSUE_PER_ACCOUNT_CERTIFICATES(). The return means issuance was accepted. Later calls do nothing. Setup then reads the route check’s CERTIFICATES stage once. SUCCESS means the certificates are already confirmed. IN_PROGRESS, or no row yet, is what that stage reports until an agent connects. The agent starts after setup returns, so setup doesn’t wait. It logs that state and continues. The agent confirms the certificates once it bootstraps. See SYSTEM$ISSUE_PER_ACCOUNT_CERTIFICATES.

Reusing a proxy your role can already see doesn’t create the proxy and doesn’t reissue certificates. Issuance requires OWNERSHIP. Setup warns when that proxy’s CERTIFICATES stage isn’t SUCCESS.

Start the agent¶

When the tunnel objects are ready, setup prints this command:

docker run -d --name dcp-agent --restart unless-stopped \
  -v /etc/dcp-agent/secrets:/etc/dcp-agent/secrets:ro \
  snowflakedb/dcp-client:latest

The image is snowflakedb/dcp-client:latest. The agent reads /etc/dcp-agent/secrets/dcp-bootstrap-token. The CLI mounts the secrets directory, so a token replaced by rename is visible to the running agent. Store the token, set file permissions, rotate it, and publish metrics with Set up Data Connectivity Proxy. That page mounts the token file and publishes a metrics port. Both mounts work.

The agent host must resolve the source hostname and open the source port. It also needs outbound access to Snowflake on port 443. The account network policy must allow the agent host’s public egress IP. A missing IP surfaces as token validation failed.

Before you start the agent, DESCRIBE DATA CONNECTIVITY PROXY reporting AGENT_HEALTH = DOWN is expected. After the agent is up, expect:

• AGENT_HEALTH = HEALTHY
• AGENT_STATUS = DCP_AGENT_LIFECYCLE_CONNECTED
• BOOTSTRAP_STATUS = DCP_AGENT_BOOTSTRAP_SUCCEEDED
• DATA_PATH_STATUS = DCP_AGENT_DATA_PATH_HEALTHY

AGENT_HEALTH = HEALTHY means the agent is connected to Snowflake. It doesn’t mean traffic reaches the source. Confirm that with the route check. Every stage must be SUCCESS. Prefix the function with the metadata database, as the CLI prints it. An unqualified INFORMATION_SCHEMA.DCP_CLIENT_ROUTE_CHECK doesn’t resolve when the session has no current database.

SELECT STAGE, STATUS, REASON
FROM TABLE(<METADATA_DB>.INFORMATION_SCHEMA.DCP_CLIENT_ROUTE_CHECK(
  AGENT_ID => '<PROXY_NAME>',
  DESTINATION => '<host>:<port>'));

See Check route reachability.

Read the CERTIFICATES stage before the agent is running. Later stages stay IN_PROGRESS until the agent is up, so they aren’t a certificate failure. After the agent is healthy, the first stage that isn’t SUCCESS is the fault: the agent, the network rule, or the source.

scai data doctor doesn’t run this check. A TCP connection from the Worker to the link-local address 169.254.0.1 isn’t evidence that a tunnel exists. That address accepts a socket when no tunnel exists.

Driver download hosts remain on the direct egress rule. A private source doesn’t remove the need for SPCS to reach those public package hosts.

Controlling cost on SPCS¶

On SPCS, the compute pool bills for every node while it runs, and the warehouse bills while the Orchestrator and Workers run queries on it. Three controls keep that spend bounded:

• A budget that suspends the warehouse and the compute pool when spend reaches a limit. You decide whether to create one.
• A compute pool sized to what the migration needs. You choose its node counts.
• Idle shutdown, which suspends the services when there’s no work. It’s always on.

Capping spend with a budget¶

A custom budget can track the migration warehouse and the compute pool together and call a stored procedure when spend reaches a threshold. Use that procedure as a stop action: it turns off auto-resume on both objects and then suspends them, so nothing restarts them until you do. Resource monitors don’t cover compute pools, so they can’t replace the budget on SPCS.

When you deploy on SPCS with the Snowflake AIM Agent for Data Warehouses, the agent offers this budget before it starts the services for the first time. It runs nothing until you agree, and it continues without a cap if you decline.

Prompt:

Cap the migration's spend at 200 credits a month.

The agent asks for the monthly limit in credits, the threshold (default 100% of the limit), and whether to trigger on actual or projected spend. It checks whether the warehouse or the pool is already in another budget and asks before it moves either object to the new budget. It lists any missing privileges as GRANT statements for an administrator, shows the SQL before it runs it, and reads the budget back afterward.

To create the same guardrail yourself, first check that neither object is in another budget. Each call returns [] when no budget tracks the object:

SELECT SYSTEM$SHOW_BUDGETS_FOR_RESOURCE('WAREHOUSE', '<warehouse>');
SELECT SYSTEM$SHOW_BUDGETS_FOR_RESOURCE('COMPUTE_POOL', '<compute_pool>');

Then run the following as the role that you want to own the budget. Use the warehouse that the Orchestrator and the Workers run on, not the warehouse of your current session. If they use different warehouses, add each one. <POOL_ID> is the pool name in uppercase, with every character other than letters, digits, and _ replaced by _. These are the names that the agent uses:

CREATE SCHEMA IF NOT EXISTS SNOWCONVERT_AI.BUDGETS;

-- The budget and its monthly limit in credits
CREATE SNOWFLAKE.CORE.BUDGET IF NOT EXISTS SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET();
CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!SET_SPENDING_LIMIT(<limit>);

-- The two objects it tracks
CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!ADD_RESOURCE(
  SELECT SYSTEM$REFERENCE('WAREHOUSE', '<warehouse>', 'SESSION', 'APPLYBUDGET'));
CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!ADD_RESOURCE(
  SELECT SYSTEM$REFERENCE('COMPUTE_POOL', '<compute_pool>', 'SESSION', 'APPLYBUDGET'));

-- The stop action. Auto-resume goes off first so that nothing restarts either object.
-- It attempts all four changes, skips a suspend only for an object that is already
-- suspended, and fails if any change failed, so that Snowflake retries the action.
CREATE OR REPLACE PROCEDURE SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_STOP()
RETURNS STRING
LANGUAGE SQL
EXECUTE AS OWNER
AS
$$
DECLARE
  failed BOOLEAN DEFAULT FALSE;
  obj_state STRING;
  stop_failed EXCEPTION (-20001, 'DMV budget stop failed: <warehouse> or <compute_pool> might still be running.');
BEGIN
  BEGIN
    ALTER WAREHOUSE <warehouse> SET AUTO_RESUME = FALSE;
  EXCEPTION
    WHEN OTHER THEN failed := TRUE;
  END;
  BEGIN
    ALTER COMPUTE POOL <compute_pool> SET AUTO_RESUME = FALSE;
  EXCEPTION
    WHEN OTHER THEN failed := TRUE;
  END;
  BEGIN
    obj_state := NULL;
    SHOW WAREHOUSES LIKE '<warehouse>';
    SELECT "state" INTO :obj_state FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
      WHERE UPPER("name") = UPPER('<warehouse>');
    IF (obj_state IS DISTINCT FROM 'SUSPENDED') THEN
      ALTER WAREHOUSE <warehouse> SUSPEND;
    END IF;
  EXCEPTION
    WHEN OTHER THEN failed := TRUE;
  END;
  BEGIN
    obj_state := NULL;
    SHOW COMPUTE POOLS LIKE '<compute_pool>';
    SELECT "state" INTO :obj_state FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
      WHERE UPPER("name") = UPPER('<compute_pool>');
    IF (obj_state IS DISTINCT FROM 'SUSPENDED') THEN
      ALTER COMPUTE POOL <compute_pool> SUSPEND;
    END IF;
  EXCEPTION
    WHEN OTHER THEN failed := TRUE;
  END;
  IF (failed) THEN
    RAISE stop_failed;
  END IF;
  RETURN 'DMV warehouse and compute pool suspended by budget';
END;
$$;

-- The SNOWFLAKE application calls the procedure. Grant again whenever you replace it.
GRANT USAGE ON DATABASE SNOWCONVERT_AI TO APPLICATION SNOWFLAKE;
GRANT USAGE ON SCHEMA SNOWCONVERT_AI.BUDGETS TO APPLICATION SNOWFLAKE;
GRANT USAGE ON PROCEDURE SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_STOP() TO APPLICATION SNOWFLAKE;

-- Call the stop when actual spend reaches 100% of the limit.
CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!ADD_CUSTOM_ACTION(
  SYSTEM$REFERENCE('PROCEDURE', 'SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_STOP()', 'SESSION', 'USAGE'),
  ARRAY_CONSTRUCT(),
  'ACTUAL',
  100);

The role needs these privileges:

To confirm that the budget can call the stop, run CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!CONFIRM_CUSTOM_ACTIONS_ACCESS(); and check that IS_VALID is TRUE. Keep the budget in its own schema, as here, so that cleaning up the migration schemas never drops it. For the budget methods, see Custom budgets and Custom actions for budgets. The budget itself incurs a small serverless compute cost; see Understanding budget costs.

Before you rely on the budget, know how it behaves:

• It’s a backstop, not a meter. A budget refreshes up to every 6.5 hours, so spend can pass the threshold before the stop runs. See Adjusting the budget refresh interval.
• Actions run at most once per period. An ACTUAL action runs at most once a month, and a PROJECTED one at most once a day. After a stop and a recovery in the same month, an ACTUAL stop doesn’t run again until the next month. Snowflake retries a failed action once, so check the state of both objects after a budget notification.
• It stops everything on those objects. Suspending the warehouse affects every other workload that uses it, and suspending the pool suspends every service in it. Use a dedicated warehouse if the cap must stop only the migration.
• Running queries aren’t canceled. Suspending a warehouse lets its running queries finish.

After a stop, raise the limit if the migration needs more, turn auto-resume back on, and then resume the services as described in Resuming services before the next wave:

CALL SNOWCONVERT_AI.BUDGETS.DMV_<POOL_ID>_BUDGET!SET_SPENDING_LIMIT(<new_limit>);
ALTER WAREHOUSE <warehouse> SET AUTO_RESUME = TRUE;
ALTER COMPUTE POOL <compute_pool> SET AUTO_RESUME = TRUE;

Sizing the compute pool for cost¶

Every node counted in MIN_NODES bills whenever the pool runs, even while the services are idle. Create the pool with MIN_NODES = 1, and use MAX_NODES as the cost ceiling:

• Services that scai creates: keep MAX_NODES = 1 unless they don’t fit. scai data orchestrator setup and scai data worker setup request cpu: 0.1 and memory: 2Gi for each service and set no limits, so a node holds as many of them as fit in its memory at 2Gi each.
• Workers whose resource requests you raised: set MAX_NODES to 2 or 3, enough nodes for the Orchestrator and two or three Worker instances. A pool wider than the source system can feed adds cost, not throughput.

Don’t set a large MIN_NODES for a heavy wave. Raise MAX_NODES and the Worker’s MAX_INSTANCES for that wave, and lower them again afterward. For instance families and per-instance sizing, see Compute pool sizing.

Idle shutdown¶

The Orchestrator and every Worker shut themselves down after a period without work. The period is DM_IDLE_SHUTDOWN_MINUTES, 60 minutes by default, and each process reads its own value. You can’t turn idle shutdown off. For the settings, see Idle shutdown settings.

• The Orchestrator shuts down when it has had no active, pending, or upcoming work for the whole period. On SPCS, it first suspends the one Worker service named in DM_SPCS_SUSPEND_AGENT_SERVICE, and then suspends its own service. scai data orchestrator setup sets that variable to DATA_EXCHANGE_WORKER_SERVICE, or to the name that you pass with --worker-service-name.
• Each Worker shuts down when it has pulled no tasks and held none for the whole period. On SPCS, it suspends its own service. On your own hosts or in Kubernetes, the Worker process exits, and your process manager decides whether to restart it.

A Worker takes its period from DM_IDLE_SHUTDOWN_MINUTES, or from [idle_shutdown] minutes in its TOML when the variable is unset. scai sets neither, so every service it creates uses 60 minutes. To change the period of an SPCS service, add the variable to its service specification.

The compute pool’s AUTO_SUSPEND_SECS is a separate, later step. The pool suspends only when no service is running in it, so AUTO_SUSPEND_SECS starts counting after idle shutdown has suspended every service in the pool. Idle shutdown stops the services; AUTO_SUSPEND_SECS then stops the pool’s nodes from billing. See Compute pool sizing.

Idle shutdown with several Workers¶

DM_SPCS_SUSPEND_AGENT_SERVICE holds one service name. When you run several Worker services, the Orchestrator suspends only the one whose name you passed to --worker-service-name. Every other Worker service is suspended only by its own idle shutdown.

With several Workers, keep these points in mind:

• The other Workers depend on their own period. A Worker whose DM_IDLE_SHUTDOWN_MINUTES is longer keeps running after the Orchestrator has suspended, and keeps the pool from suspending.
• A failed self-suspend has no fallback for most Workers. If a Worker can’t suspend its own service, it stays up and retries after 1, 2, 4, 8, and 16 minutes, then waits for another full idle period. The Orchestrator still suspends the Worker named in --worker-service-name, but no other.
• A Worker that you create by hand follows the same rule. The Orchestrator suspends it only when it’s DATA_EXCHANGE_WORKER_SERVICE in the migration metadata schema (SNOWCONVERT_AI.DATA_MIGRATION by default), or the service whose name you passed to --worker-service-name.

After a wave, run scai data worker list and check that every Worker is SUSPENDED. Suspend any that isn’t with scai data worker stop --service-name <name>.

Resuming services before the next wave¶

A suspended service stays suspended until you resume it. Submitting a workflow with scai data migrate create-workflow or scai data validate create-workflow doesn’t resume it. Before the next wave, resume the Orchestrator and every Worker service, including the ones that suspended themselves:

scai data orchestrator start
scai data worker start --service-name <worker_service_name>

• Without --service-name, scai data worker start resumes DATA_EXCHANGE_WORKER_SERVICE.
• If the compute pool has suspended too, for example after AUTO_SUSPEND_SECS or a budget stop, add --resume-compute-pool. Without it, the command fails while the pool is suspended.
• A workflow that you submit while its Workers are suspended stays pending until you resume them.

Related content¶

• Data Migration & Validation overview
• Manual SPCS worker setup
• Data migration
• Data migration advanced configuration
• Data validation
• Data migration configuration reference
• Data migration (CLI)
• The SNOWCONVERT_AI database