Make ML predictions human-interpretable with a Cortex Agent¶
Turn a Model Registry prediction into an explanation people can use: why the model scored a member high, what each driver means, and how much to trust the score.
| Stage | Build, Evaluate |
| Topics | Cortex Agents |
| Status | Preview |
| Time to complete | 45min |
| Edition | Any edition with Cortex Agents |
| Regions | Any region with Cortex Agents; warehouse Python with the Snowflake Anaconda channel |
| Privileges | ACCOUNTADMIN for sql/01_setup.sql and sql/99_teardown.sql; 02 to 07 run as COOKBOOK_EXPLAIN_MODEL_PREDICTIONS_ADMIN |
| Related skills | machine-learning (bundled with Cortex Code), agent-studio (bundled with Cortex Code) |
| Last verified | 2026-11-10 |
| Authors | Chris Caudill |
| Code | View all files or download explain-model-predictions.zip, which includes this page as README.md. Teardown: sql/99_teardown.sql |
This recipe is a reference implementation: review the code and adapt it to your environment before production use.
A model score on its own tells a care manager nothing about why. This recipe
turns each prediction into a plain-language explanation: what drove the score,
what each driver means, and how much to trust it. Ask the agent “why was member
M000003 flagged as a likely high-cost claimant?” and it answers with the score,
how often members with similar scores really were high-cost, and the three
things that pushed the score up most, each compared with a typical member and
described in your plan’s own terms. Underneath, a classifier in the Snowflake
Model Registry supplies the prediction and its Shapley values through
PREDICT_PROBA and EXPLAIN, a feature dictionary you control supplies what
each feature means, and holdout calibration supplies the confidence. The agent
is instructed never to present a driver as a cause.
When to use this¶
- A classifier already drives a workflow (care management outreach, fraud review, credit decisions) and the people acting on its scores ask “why this one?”.
- You need answers grounded in the model’s own attributions rather than an LLM’s guess about what usually drives the outcome.
- You need to report confidence honestly: what a score of 0.74 meant on members the model never saw, not the raw probability.
When not to use it:
- You need to know what will happen if you intervene (“will outreach lower this member’s cost?”). Shapley values describe the model, not the world. Use a causal design such as the causally grounded agent.
- The model is PyTorch, TensorFlow, or a Hugging Face pipeline. The registry’s
EXPLAINmethod covers scikit-learn, XGBoost, LightGBM, and CatBoost only. - The model only runs on Snowpark Container Services.
EXPLAINruns in a warehouse.
How it works¶
Key concepts:
- Explainability is decided when you log the model.
log_modelcreates anEXPLAINmethod when the model type supports it,sample_input_datais given, and the target platforms includeWAREHOUSE. This recipe also setsoptions={"enable_explainability": True}explicitly. A version cannot gain explainability later; you log a new version. - Shapley values are relative to background data. The rows passed as
sample_input_data(500 training members here) define the “average” member each explanation is measured against. For this XGBoost classifier the values are log-odds contributions:LN(p / (1 - p))equals the base value (about -2.10, a score of 0.11) plus the sum of the nine contributions, to six decimal places on every holdout row. - A score is not a confidence. 04 scores the 20% holdout and records, per score band, how often members in that band actually were high-cost. The agent reports that observed rate and, when asked how confident it is, the holdout AUC and the precision at the flag threshold.
- Meaning comes from a table, not from the model’s general knowledge.
EXPLAINreturns a number per column name; it says nothing about whatOPEN_CARE_GAPSmeans.FEATURE_DICTIONARYholds, per feature, a label, a unit, a definition (what it measures and how it is counted), andHOW_TO_EXPLAINguidance (what it is a signal of and what not to say). The tools return those fields with every driver, a third tool returns the whole dictionary, and the instructions tell the agent to define features only from them and to say when a question goes beyond them. Each tool result stays around 5 KB. - Least privilege through caller’s rights. The user role gets USAGE on the agent, the three procedures, and the model, and SELECT on four tables. USAGE on a model allows inference without exposing its internals; READ would also expose metadata. There is no Cortex Analyst tool, so the agent cannot compute its own numbers from raw rows.
Prerequisites¶
| Requirement | Notes |
|---|---|
| A region with Cortex Agents | The orchestration model is auto. Change it in 06_agent.sql if your account restricts models. |
ACCOUNTADMIN for 01_setup.sql and 99_teardown.sql | 01 creates the two roles, the warehouse, and the database, and grants SNOWFLAKE.CORTEX_USER. 02 to 06 run as COOKBOOK_EXPLAIN_MODEL_PREDICTIONS_ADMIN; 07 runs as COOKBOOK_EXPLAIN_MODEL_PREDICTIONS_USER. |
| Warehouse Python 3.11 with the Snowflake Anaconda channel | 03 trains in a procedure using snowflake-ml-python, xgboost, scikit-learn, and pandas. No local Python environment and no artifact repository are needed. |
Snowflake CLI (snow sql) | Each script runs as written with snow sql -f. |
Model Registry explainability is documented as a preview feature, so this recipe is marked preview.
Quickstart¶
From explain-model-predictions/:
snow sql -f sql/01_setup.sql: roles, warehouse, database, schema.snow sql -f sql/02_data.sql: 20,000 synthetic members (11% high-cost), a 20% holdout, and the feature dictionary with definitions.snow sql -f sql/03_train_and_register.sql: train XGBoost and logHIGH_COST_CLAIMANTV1 withEXPLAIN. Takes a few minutes.snow sql -f sql/04_calibrate.sql: score the holdout, writeSCORE_CALIBRATIONandMODEL_CARD, and check that the explanations are in log-odds.snow sql -f sql/05_tools.sql: the three tool procedures, plus four test calls.snow sql -f sql/06_agent.sql: the agent and the user role’s grants.snow sql -f sql/07_ask_agent.sql: five questions to the agent as the user role, then the raw tool output.snow sql -f sql/99_teardown.sql.
Walkthrough¶
Logging the model with explainability¶
background is a Snowpark DataFrame, not pandas, so the registry also records
lineage back to MEMBER_FEATURES. The trees are shallow and regularized
(max_depth=3, min_child_weight=20). A deeper first attempt gave the
pure-noise PORTAL_LOGINS_12M more attribution than two real features, and an
explanation built on that model would have told care managers that portal logins
matter.
Calling EXPLAIN from SQL¶
EXPLAIN is a table function, so it goes in FROM with a lateral join, one
output column per feature named <FEATURE>_EXPLANATION:
The documented WITH mv AS MODEL ... VERSION V1 alias works in a single SELECT,
but not from a later CTE (“Unknown table function MV!EXPLAIN”). The tools use the
inline MODEL(<name>, V1)! form, which works anywhere.
Turning columns into ranked drivers¶
SCORE_AND_EXPLAIN_MEMBER packs the nine contributions into an object,
flattens it, joins the feature dictionary, and keeps the top five by absolute
value:
Each driver in the result carries label, unit, definition,
how_to_explain, member_value, population_median, effect_on_score (raises
or lowers), shapley_logodds, and share_of_total_attribution. The share is
what the agent uses to compare drivers; it is not converted into percentage
points of probability, because log-odds contributions do not add up in
probability space.
The feature dictionary¶
The model only knows column names, and an LLM asked to explain
CHRONIC_CONDITIONS or OPEN_CARE_GAPS will fill in a plausible meaning from
general knowledge, which may not match how your plan counts them.
FEATURE_DICTIONARY in 02_data.sql pins the meaning down:
DEFINITION is what the feature measures and how it is counted, including
look-back windows and, for counts, what is counted (the seven chronic conditions
on a fixed list). HOW_TO_EXPLAIN is guidance for the agent: what the feature
is usually a signal of, and the wrong conclusion to avoid. For the noise feature
it says to call any weight “a minor statistical effect of the model”, so an
imperfect model does not produce a confident wrong reason.
The definitions describe this synthetic data. When you point the recipe at real
features, write the dictionary from your data documentation: the agent repeats
it, so it is where your plan’s terminology and review rules live.
DESCRIBE_FEATURES() returns the whole dictionary, which lets the agent answer
“what counts as a care gap?” without a member ID.
What-if without dynamic SQL¶
WHAT_IF_MEMBER checks the feature name against FEATURE_DICTIONARY and the
value against that row’s range, then swaps the one column with IFF:
Nothing the user or the model writes is concatenated into SQL. The result is labeled model sensitivity: the score’s response to one input with everything else held fixed. A member’s ER visits cannot drop to zero while everything else stays the same, so the instructions forbid presenting it as an intervention effect.
The agent instructions¶
The response instructions fix the shape of an explanation (score, then what the
score meant on holdout members, then three drivers against the median, each with
a clause from its how_to_explain) and the words: drivers “contributed to” or
“pushed up” the score; AUC measures ranking across all members, not certainty
about one. Features are named by label and defined only from tool results, and
the agent says when a question goes beyond them (for example which conditions a
member has, when the model only sees the count). how_to_explain is addressed
to the agent, so the instructions also tell it not to read that guidance back to
the user as instructions; without that line, a test answer did. The
orchestration instructions also tell the agent not to use the built-in chart
tool, which every agent has.
Verify¶
-
02returns 20,000 members, a high-cost rate of 0.111, and 4,001 holdout members. The data is generated from hashes, so these numbers are the same on every run. -
03returnsfunctions: ["EXPLAIN", "PREDICT", "PREDICT_PROBA"]and a holdout AUC of 0.8446.SHOW FUNCTIONS IN MODEL ... VERSION V1listsEXPLAINas a table function. -
04returns aBASE_VALUE_SPREADof about 1e-6 and six calibration bands. The ‘very low’ band (under 0.05) holds 2,326 holdout members with an observed rate of 0.028; the ‘very high’ band (0.50 and above) holds 205 with an observed rate of 0.698. The middle bands run slightly hot: ‘elevated’ has a mean score of 0.259 against 0.185 observed, which is why the agent quotes the observed rate.MODEL_CARDshows a base value of -2.094 log-odds (a score of 0.11), and precision 0.56 and recall 0.45 at the 0.35 flag threshold. -
05’s first call returns member M000003 with a score of 0.74, band ‘very high’, and drivers led by chronic conditions (5 against a median of 1, 41% of the attribution) and ER visits (5 against 0, 32%). The second call, ER visits set to 0, returns 0.324 in ‘elevated’. The third returns anerrorlisting the allowed features. The fourth,DESCRIBE_FEATURES(), returns nine features, each with a definition and explanation guidance. Each result is about 5 KB. -
07returns five rows:- 1 and 2 call
score_and_explain_member. The answers quote the score and the band’s observed rate, name the drivers by label against medians with a short note on what each signals, and describe AUC as a ranking measure. - 3 calls
what_if_memberwithER_VISITS_12Mand0and calls the result model sensitivity. - 4 calls
describe_featuresand defines care gaps and prior-year allowed cost from the dictionary, in plain language. - 5 asks which chronic conditions the member has. The agent says the model only sees the count (5 of the 7 listed conditions) and points to the claims history for names.
None says a feature causes high cost. Agent wording varies between runs; the numbers and definitions in it come from the tools and do not.
- 1 and 2 call
-
DATA_AGENT_RUNruns under the session role that07sets withUSE ROLE, even when the user’s default role is PUBLIC.
Gotchas¶
| Symptom | Cause | Fix |
|---|---|---|
The version has no EXPLAIN method | Logged with signatures instead of sample_input_data, with an SPCS-only target, or with explainability off | Log a new version with sample_input_data, a WAREHOUSE target, and enable_explainability: True. Existing versions cannot be changed. |
| “Unknown table function MV!EXPLAIN” | A WITH mv AS MODEL ... alias referenced from another CTE | Use TABLE(MODEL(<db.schema.model>, V1)!EXPLAIN(...)) inline |
| Contributions do not add up to the probability | XGBoost explanations are in log-odds | Compare LN(p / (1 - p)) with the base value plus the sum; 04 checks this |
| An irrelevant feature appears among the drivers | The model fit noise, and the explanation is faithful to the model | Regularize or remove the feature. Explanations show what the model uses, not what is true. Its HOW_TO_EXPLAIN can tell the agent to call it a minor statistical effect, as PORTAL_LOGINS_12M’s does |
| The agent defines a feature differently from how your plan counts it | The dictionary is missing the feature or its DEFINITION is vague, so the model falls back on general knowledge | Write the definition from your data documentation, including look-back window and what is counted; rerun 02 and 06 |
| An answer tells the user “say…” or “do not say…” | The agent read HOW_TO_EXPLAIN back as text for the user | Keep the instruction in 06_agent.sql that the guidance is for the agent and must not be repeated as instructions |
CREATE PROCEDURE fails with “unexpected ‘COMMENT’” | COMMENT placed after EXECUTE AS | Put COMMENT before EXECUTE AS CALLER |
| “Function MAX does not support ARRAY argument type” | Pivoting an ARRAY column with MAX(IFF(...)) | Build one object per row and combine them with OBJECT_AGG, as WHAT_IF_MEMBER does |
| The tool calls fail with “does not exist or not authorized” for the user role | 02 to 04 recreated the tables or the model, which drops their grants | Rerun 06_agent.sql after rerunning any of 02 to 04 |
| The agent draws a chart | Every agent has a built-in chart tool | The orchestration instructions forbid it; filter in the client as well |
| A custom tool rejects an OBJECT or VARIANT argument | Custom tools take scalar parameters | Pass scalars, as these tools do, or a JSON string parsed inside the procedure |
snow sql echoes if MODEL_NAME in : for the procedure body | The CLI’s display treats [...] as markup | Display only; the statement sent to Snowflake is intact |
Teardown¶
sql/99_teardown.sql runs as ACCOUNTADMIN and drops the database (model and
versions, agent, procedures, tables), the warehouse, and both roles, then lists
anything left with a COOKBOOK_EXPLAIN_MODEL_PREDICTIONS prefix. No
account-level settings are changed by this recipe.
Run it with a coding agent¶
Related skills¶
machine-learning(bundled with Cortex Code): its model-registry sub-skill covers logging other model types, explainability options, and visualizing Shapley values withsnowflake.ml.monitoring.explain_visualize.agent-studio(bundled with Cortex Code): build an evaluation set for this agent and check that answers keep the score, the band rate, and the non-causal wording.