Parallel hyperparameter tuning as a multi-node ML Job¶
Run an XGBoost hyperparameter search across a multi-node compute pool with snowflake.ml.modeling.tune, log the trials as an experiment, and register the best model.
| Stage | Build, Evaluate |
| Topics | Snowflake ML |
| Status | GA |
| Time to complete | 30min |
| Edition | Any edition with Snowpark Container Services |
| Regions | Any region with Snowpark Container Services compute pools; ML Jobs are not available in the People’s Republic of China |
| Privileges | ACCOUNTADMIN for sql/01_setup.sql and sql/99_teardown.sql (role, warehouse, compute pool, database); sql/02 and the notebook run as COOKBOOK_HYPERPARAMETER_TUNING_ADMIN |
| Related skills | machine-learning (bundled with Cortex Code) |
| Last verified | 2026-11-16 |
| Authors | Trace Smith |
| Code | View all files or download hyperparameter-tuning.zip, which includes this page as README.md. Teardown: sql/99_teardown.sql |
This recipe is a reference implementation: review the code and adapt it to your environment before production use.
Run a whole XGBoost hyperparameter search on Snowflake compute instead of a laptop or a single notebook kernel. One script, submitted as a Snowflake ML Job, spreads two dozen training trials across every CPU on two compute pool nodes with snowflake.ml.modeling.tune. The job registers the best model in the Model Registry with its hyperparameters and metrics, and the notebook records every trial as a run in Snowflake Experiment Tracking so the search can be reviewed and compared later.
When to use this¶
- A single model trains in seconds to minutes on one node, and you want to try many configurations without waiting for them one after another.
- You want the search to be reproducible and asynchronous: submitted from any Python environment, running close to the data, and reattachable later by job ID.
- You want the tuning history (every configuration and its metrics) kept in Snowflake next to the model it produced.
When not to use it:
- One model is too large to train on a single node. Train it with a distributed trainer first; see Distributed training as a multi-node ML Job. The two combine: call
XGBEstimatorinsidetrain_funcand setuses_snowflake_trainer=True. - You need retraining on a schedule with a promotion gate. See Scheduled retraining with a task graph and a quality gate, and swap its training step for this job when the search outgrows a warehouse.
- The search space is tiny. A loop in a notebook is simpler than a job.
How it works¶
- The Tuner does the scheduling. You write one
train_funcand declare a search space.Tuner.run()samples configurations, runstrain_funconce per trial on whichever worker has free CPUs, and collects what each trial reports throughget_tuner_context().report(). - Parallelism is two numbers.
target_instancessets how many nodes join the job’s Ray cluster.max_concurrent_trialssets how many trials run at once on each node. The training code does not change when you scale either one. - The search runs as an ML Job.
submit_fileuploads the script to a stage and starts a job service on the compute pool. The script’s__return__value comes back throughjob.result(), and the job can be reattached withjobs.get_job(job_id)after the notebook closes. - The job closes the loop itself. It logs the best model to the Model Registry from inside the container, so there is no artifact to download. The notebook then logs every trial to an experiment for review in Snowsight.
Prerequisites¶
| Requirement | Notes |
|---|---|
| Snowpark Container Services in your region | ML Jobs run on a compute pool. ML Jobs are not available in the People’s Republic of China. |
ACCOUNTADMIN | For sql/01_setup.sql (creates a role, warehouse, compute pool, and database) and sql/99_teardown.sql. sql/02 and the notebook run as COOKBOOK_HYPERPARAMETER_TUNING_ADMIN. |
Compute pool quota for 3 CPU_X64_M nodes | sql/01 creates the pool with MAX_NODES = 3; the job uses 2. The pool bills per active node and suspends after 10 idle minutes. |
| Python 3.11+ locally | pip install -r notebooks/requirements.txt, plus Jupyter or VS Code. snowflake-ml-python 1.26.0 or later is required for ML Jobs. |
| A Snowflake connection for your own user | The notebook uses the active Snowpark session, or Session.builder.getOrCreate() with SNOWFLAKE_DEFAULT_CONNECTION_NAME. |
Quickstart¶
Object names are fixed. Besides the warehouse size, you may change the pool’s INSTANCE_FAMILY and MAX_NODES in sql/01, and the search settings (TARGET_INSTANCES, NUM_TRIALS, MAX_CONCURRENT_TRIALS, THREADS_PER_TRIAL) in the notebook’s configuration cell. Keep TARGET_INSTANCES at or below MAX_NODES.
sql/01_setup.sqlasACCOUNTADMIN: the role, warehouse, compute pool, database,MLschema, andML.JOBS_STAGE.sql/02_data.sql: buildsML.LOAN_APPLICATIONS(1M rows) and prints its row count and default rate.notebooks/01_hyperparameter_tuning.ipynb: run every cell. The job takes several minutes, longer on the first run while the pool starts its nodes.sql/99_teardown.sqlasACCOUNTADMIN.
Walkthrough¶
Setup (sql/01)¶
An ML Job needs USAGE on the compute pool, CREATE SERVICE on the schema it runs in, and access to a stage for its payload. Owning the database gives the recipe role CREATE SERVICE, CREATE STAGE, CREATE MODEL, and CREATE EXPERIMENT on the ML schema, so the role needs nothing at account level beyond the pool and warehouse grants. MONITOR on the pool lets the role see the pool’s state.
The trial function (notebooks/src/tune_xgb.py)¶
The train and validation splits are DataConnectors over a deterministic hash split of LOAN_ID, so the notebook can rebuild the same hold-out set. Trials train open-source XGBoost, so uses_snowflake_trainer=False tells the Tuner to pack several single-process trials onto each node. The script accepts --table only as an unquoted DATABASE.SCHEMA.TABLE, because the name is placed into the split query.
Submitting the job (notebook step 3)¶
Search settings travel as command-line arguments, so the budget changes without editing the script. On CPU_X64_M the notebook’s defaults run 3 trials of 2 threads each per node, 6 at a time across 2 nodes. Raise TARGET_INSTANCES (up to the pool’s MAX_NODES) to finish the same search sooner.
Reviewing and using the result (notebook steps 5 to 7)¶
job.result() returns every trial’s configuration and validation metrics. The notebook plots search progress and each hyperparameter against validation ROC AUC, logs each trial with ExperimentTracking.start_run, log_params, and log_metrics, and scores ten rows with the registered model in the warehouse with ModelVersion.run(..., function_name="predict_proba").
Verify¶
-
sql/02returnsn_rows = 1000000and adefault_ratebetween 0.15 and 0.20. -
Notebook step 1 prints
COOKBOOK_HYPERPARAMETER_TUNING_ADMIN,COOKBOOK_HYPERPARAMETER_TUNING_WH, andML. -
Step 4 prints
Final status: DONE, and the key log lines includeRay cluster: 2 node(s)andTuning finished: 24 trials. -
Step 5 shows 24 trials. The best validation ROC AUC is well above 0.5, and the spread between best and median is small.
-
Step 6 prints
Logged 24 runs to experiment COOKBOOK_HYPERPARAMETER_TUNING.ML.LOAN_DEFAULT_HPO. -
Step 7 prints the registered metrics and ten rows with a
DEFAULT_PROBABILITYbetween 0 and 1. -
In SQL, as the recipe role:
The first returns one version per notebook run, and the second lists
LOAN_DEFAULT_HPO.
Gotchas¶
| Symptom | Cause | Fix |
|---|---|---|
The job stays PENDING for several minutes on the first run | The compute pool starts suspended and is provisioning nodes | Wait; check DESCRIBE COMPUTE POOL COOKBOOK_HYPERPARAMETER_TUNING_POOL for ACTIVE or IDLE |
| The job logs “Compute pool busy (2/3 nodes in use, 2 nodes required)” and waits | The pool is still adding nodes, or another job on the same pool holds them | Informational; the job starts once enough nodes are free. Check SHOW SERVICES IN COMPUTE POOL COOKBOOK_HYPERPARAMETER_TUNING_POOL |
| The job fails waiting for instances | target_instances is above the pool’s MAX_NODES, or the account’s node quota is exhausted | Keep TARGET_INSTANCES at or below MAX_NODES, or lower it |
submit_file fails with insufficient privileges on the compute pool or schema | The session is not using the recipe role | Run notebook step 1, which calls session.use_role(ROLE) |
ImportError for snowflake.ml.jobs or snowflake.ml.experiment | Old snowflake-ml-python | pip install -r notebooks/requirements.txt (ML Jobs need 1.26.0 or later) |
| Trials run one at a time | max_concurrent_trials times threads_per_trial exceeds a node’s schedulable CPUs | Lower either value; a CPU_X64_M node fits 3 trials of 2 threads |
parser.error: --table must be an unquoted DATABASE.SCHEMA.TABLE name | A quoted or two-part name was passed | Pass the fully qualified, unquoted name |
| Integer hyperparameters arrive as floats | Search spaces sample numbers; the trial casts them | Keep the int(...) casts in train_func |
Teardown¶
Run sql/99_teardown.sql as ACCOUNTADMIN. It stops everything running on the compute pool and drops the pool, then drops the database, which removes the table, the payload stage, the job services, the model and its versions, and the experiment with its runs. Last, it drops the warehouse and the role. Every name comes from the script’s SET block, which matches setup, and the script ends with SHOW statements that should all return no rows.
Run it with a coding agent¶
Related skills¶
machine-learning(bundled): training, registering, deploying, and monitoring models on Snowflake, including ML Jobs. This recipe covers one end-to-end tuning pattern; that skill covers the rest of the ML workflow and ad-hoc modeling questions.