Scoping agentic implementations: from ‘talk to your data’ to a testable agent spec¶

A workshop method for turning a broad agent idea into personas, a question taxonomy, a registered evaluation dataset, and a phased delivery plan.

This recipe is a reference implementation: review the code and adapt it to your environment before production use.

Most agent projects stall on scope, not technology. “Talk to your data” is not a requirement anyone can build or test. This recipe is a workshop method that starts with a one-page view of the platform and turns a broad idea into a scoped agent specification: who the agent serves, what they will ask, which Snowflake primitive answers each kind of question, what the agent must refuse, and how you will know it works. The output is a written spec plus a registered Cortex Agent evaluation dataset, so the first build of the agent can be measured on day one. A small SQL lab supports the workshop: it expands a seed question taxonomy with AI_COMPLETE, turns it into an evaluation dataset, and, after launch, classifies real user questions back into the taxonomy.

When to use this¶

• A first architecture conversation about Cortex Agents, before anyone has picked a use case: answering “what do we build and what does Snowflake give us?” and reviewing data movement with a security team.
• Kicking off an agent project with a business owner, an analytics engineer, and a technical lead in the room.
• An agent pilot exists but nobody can say whether it is good enough to expand.
• You need an evaluation dataset before the agent exists, so that the first build has a baseline.

When not to use it:

• The agent is already scoped and has an evaluation dataset. Go to agent evaluations and agent CI/CD with dbt.
• You need a production multi-tenant access model. This recipe scopes the requirement; multi-tenant agents implements it.

How it works¶

Key ideas:

1. Scope by persona, then by question. One persona and two or three question categories are enough for Phase 1. Everything else is a later phase with its own exit criteria.
2. Every question category maps to exactly one primitive. Lookup and aggregation go to Cortex Analyst over a semantic view, policy questions go to Cortex Search, and out-of-scope questions go to the agent’s instructions as explicit refusals. A category with no primitive is a blocker to record, not a feature to promise.
3. The evaluation dataset comes before the agent. Writing ground truth forces the team to agree on what a correct answer is. Where the answer is a known number, state the literal value. Where it is not, write a rubric of what the answer must and must not contain.
4. Absolute dates only. “Last month” drifts every day and breaks ground truth. “Between July 1 and September 30, 2024” does not.
5. Production questions feed the next iteration. After launch, classify what users actually ask into the same taxonomy. Categories users ask about that you did not scope become new evaluation rows or a new phase.
6. Fix instructions and data models before changing architecture. Start with one agent and attach the semantic views and tools its users need. When answers are wrong, tighten the instructions and put business rules (which org counts as “Sales”, which statuses count) in the semantic view. Add an agent only when a team must own separate instructions. See multi-agent orchestration.

The platform on one page¶

Open the workshop with the architecture, before anyone picks a persona. It separates what Snowflake provides from what the team builds and shows which calls leave Snowflake, so governance and cost are part of the scope from the first conversation rather than after the pilot.

Source: diagrams/cortex-agents-reference-architecture.svg (1860 x 1270). The PNG is a 2x render.

The diagram is hub and spoke, with the agent object at the center. Solid arrows into the agent are enforced controls, dashed arrows are telemetry the agent emits, and orange arrows leave Snowflake.

1. The agent is a schema-level object. It bundles the orchestration model, instructions, and tool configuration, carries grants like any other object, and has versions and aliases, so callers never wire models, retrieval, or SQL execution themselves.
2. There are three ways in. The REST agent:run endpoint serves an application that drives the conversation, SNOWFLAKE.CORTEX.DATA_AGENT_RUN serves SQL, tasks, and procedures, and the Snowflake-managed MCP server serves an MCP client that discovers and calls tools itself; MCP or the Agents REST API covers the choice.
3. Identity is the caller’s. A run uses the calling user’s default role (the REST API can override it with a header), and every tool runs with that role’s privileges unless the tool is defined to run with its owner’s rights. The caller needs the SNOWFLAKE.CORTEX_USER or SNOWFLAKE.CORTEX_AGENT_USER database role, USAGE on the agent, and privileges on every object its tools use.
4. Tools decide what the agent can reach. Semantic views, search services, and procedures are the grounding, and an agent with no tools answers only from the model’s general knowledge. Cortex Search queries run with the service owner’s rights, so any role with USAGE on the service sees what the owner indexed.
5. Cortex serves the model and holds the conversation. The agent names a model, or auto, which Cortex serves with cross-region inference where the account’s CORTEX_ENABLED_CROSS_REGION setting allows, with no model endpoint or API key to manage. Threads hold conversation state server-side, so callers pass thread_id and parent_message_id instead of replaying history.
6. Two built-in tool types reach the public internet: web search and MCP connectors. Everything else, including model inference, runs within Snowflake unless you give a custom tool an external access integration, and an answer returned to a caller outside Snowflake leaves with the response.
7. Cost and behavior are observable and bounded per run, per agent, and per user. The usage view and monitoring events show what happened, the orchestration budget caps one run, custom budgets track spend without blocking requests, and per-user quotas set credit limits with optional blocking. AI usage and cost attribution turns the usage view into chargeback.

What you build versus what Snowflake provides¶

Data movement¶

Govern the last row at the source: what the calling role can read is what can end up in an answer.

Prerequisites¶

Quickstart¶

1. Run sql/01_setup.sql as ACCOUNTADMIN. It creates COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS and grants the lab role to you.

2. Run the workshop activities (personas, question brainstorm) using the templates in the Walkthrough.

3. Put your seed questions into sql/02_question_taxonomy.sql and run it to expand them with AI_COMPLETE.

4. Run sql/03_eval_dataset.sql to build EVAL_QUESTIONS and register SCOPING_EVAL_DATASET. Review every row marked needs_review.

5. When your Phase 1 agent exists, edit config/eval_config.yaml and upload it:

   Copy code

   snow stage copy config/eval_config.yaml @COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_CONFIGS/ \
     --overwrite --role COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS_ROLE
   

   --role matters when your default role is not the lab role: without it the upload runs as the default role, which cannot see the lab database.

   Then start a baseline run (see Walkthrough).

6. After launch, have an account administrator run the two grants in the header of sql/04_observability.sql, set its agent_* variables, and run it to compare real questions with your taxonomy.

Walkthrough¶

Activity 1: Personas (10 minutes)¶

Fill one card per persona and pick exactly one for Phase 1.

Activity 2: Question taxonomy (15 minutes)¶

Brainstorm 15 to 20 questions the Phase 1 persona would actually type, then tag each one.

The platform on one page shows what each primitive provides, what the team builds for it, and whether its calls leave Snowflake.

Push back on vague questions (“How is business?”) by asking what the person would type on Monday morning. Push back on tool-first thinking by asking which question needs the tool.

Activity 3: Expand the taxonomy with AI_COMPLETE¶

sql/02_question_taxonomy.sql stores the seeds in SEED_QUESTIONS, then makes one AI_COMPLETE call per category and one per Phase 1 lookup or aggregation question. A JSON schema keeps every response parseable and constrains risk to three values:

AI_COMPLETE(
  model => $model,
  prompt => 'You are helping build an evaluation dataset for an AI agent used by ' || $persona || '. ...'
         || 'Use absolute dates in 2024 ..., never relative dates such as "last month". ...',
  model_parameters => {'temperature': 0},
  response_format => {
    'type': 'json',
    'schema': {'type': 'object',
               'properties': {'questions': {'type': 'array', 'items': {'type': 'object',
                 'properties': {'question': {'type': 'string'},
                                'risk': {'type': 'string', 'enum': ['low', 'medium', 'high']}},
                 'required': ['question', 'risk']}}},
               'required': ['questions']}})

The prompt includes the category’s existing examples, so generated questions stay inside the scoped taxonomy. The ALL_QUESTIONS view combines seeds and synthetic questions for the team to review. With the sample seeds, one run produced 11 manual, 15 new, and 12 rephrased questions.

Activity 4: Build and register the evaluation dataset¶

A Cortex Agent evaluation dataset has two columns: INPUT_QUERY (VARCHAR) and GROUND_TRUTH (VARIANT). Answer correctness reads the ground_truth_output key; other keys are ignored by the system metrics but visible to custom metrics, so sql/03_eval_dataset.sql also stores category, expected_tool, and needs_review.

Where the answer is a known number, the script computes it from the data, so ground truth cannot drift from the table:

SELECT 'What was closed-won revenue for the West region between April 1 and June 30, 2024?',
       TO_VARIANT(OBJECT_CONSTRUCT(
         'ground_truth_output',
           'Closed-won revenue for the West region between April 1 and June 30, 2024 is $'
           || TRIM(TO_VARCHAR(SUM(revenue), '999,999,999.00')) || '. ...',
         'category', 'lookup', 'needs_review', FALSE))
  FROM SAMPLE_DEALS
 WHERE region = 'West' AND stage = 'Closed Won' AND close_date BETWEEN '2024-04-01' AND '2024-06-30'

For the other Phase 1 questions it drafts a rubric with AI_COMPLETE (what a correct answer must and must not contain, no invented numbers) and sets needs_review = TRUE. Do not gate on drafted rows until a person has read them and, where possible, replaced them with literal values.

The script then registers the dataset. SYSTEM$CREATE_EVALUATION_DATASET uses expected_tools as the mapping key for the ground truth column (the YAML dataset: block uses ground_truth):

CALL SYSTEM$CREATE_EVALUATION_DATASET(
  'Cortex Agent',
  'COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_QUESTIONS',
  'COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.SCOPING_EVAL_DATASET',
  OBJECT_CONSTRUCT('query_text', 'INPUT_QUERY', 'expected_tools', 'GROUND_TRUTH'));

The config template config/eval_config.yaml references that dataset, pins a committed agent version, and pins metric versions to v3. The stage that holds it uses a CSV file format with FIELD_DELIMITER = NONE, which evaluations require. Once your agent exists, start a baseline from the LAB schema (evaluations run in the session’s schema):

USE SCHEMA COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB;
CALL EXECUTE_AI_EVALUATION('START', OBJECT_CONSTRUCT('run_name', 'phase_1_baseline'),
  '@COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_CONFIGS/eval_config.yaml');
CALL EXECUTE_AI_EVALUATION('STATUS', OBJECT_CONSTRUCT('run_name', 'phase_1_baseline'),
  '@COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_CONFIGS/eval_config.yaml');

The role that runs it also needs EXECUTE TASK on the account, which 01_setup.sql grants to the lab role, CREATE TASK and CREATE FILE FORMAT on the schema, which the lab role has as the schema owner, and USAGE and MONITOR on the agent; see the evaluation access requirements in References. The agent CI/CD with dbt recipe turns this baseline into a gate.

Activity 5: Scope multi-tenancy (10 minutes)¶

Answer these before Phase 1, even if the answer is “single tenant”:

The lab’s ENTITLEMENTS table maps an application user to regions without a Snowflake user per person. The application sets an immutable session attribute, and a row access policy reads it with SYS_CONTEXT('SNOWFLAKE$SESSION_ATTRIBUTES', ...). The policy must fail closed: with no attribute, no rows.

CREATE ROW ACCESS POLICY DEALS_BY_ENTITLEMENT AS (row_region VARCHAR) RETURNS BOOLEAN ->
  EXISTS (SELECT 1 FROM ENTITLEMENTS e
           WHERE e.user_id = SYS_CONTEXT('SNOWFLAKE$SESSION_ATTRIBUTES', 'app_user_id')
             AND e.allowed_region = row_region);

-- No attribute set: 0 rows are visible.
CALL SET_SYS_CONTEXT('SNOWFLAKE$SESSION_ATTRIBUTES', 'app_user_id', 'user_003');
-- Now only the East and Northeast rows are visible.

Record in the spec that agent evaluations cannot pass session attributes, so a tenant-filtered agent needs a separate test plan. Multi-tenant agents covers the full pattern, including the REST variables payload.

Snowflake CoWork profiles¶

Profiles are in preview and not yet generally available. A profile is a Cortex Agent that defines what CoWork users see. Like any agent, it can hold semantic views, search services, and tools itself, or route questions to other agents linked to it.

Start with one profile for everyone, with the semantic views, search services, and tools its users need attached directly. Add only what a trigger requires:

With linked agents, keep their scopes from overlapping, keep views that are joined together in one agent, put generic tools (charting, web search, connectors) on the profile and never the same skill or connector on both layers, and test routing with real questions for each agent before rollout.

The agent spec¶

Synthesize the activities into one document. Treat it as the contract between the business goal and the build.

AGENT SPECIFICATION
Agent name:        <db.schema.agent>
Owner:             <team and accountable person>
Date:              <YYYY-MM-DD>

PERSONAS
  Phase 1: <persona, frequency, success signal>
  Later:   <persona, phase>

QUESTION TAXONOMY
  | Category     | Example (absolute dates)          | Primitive       | Phase |
  | lookup       | ...                               | Cortex Analyst  | 1     |
  | out_of_scope | ...                               | Instructions    | 1     |

TOOLS (Phase 1)
  Semantic view:  <name, tables, readiness>
  Instructions:   <refusal boundaries>

OUT OF SCOPE
  <explicit list the agent declines>

SUCCESS METRICS
  Business:  <for example, 80% of pilot questions answered without a ticket>
  Quality:   answer_correctness (v3) >= <threshold>, logical_consistency (v3) >= <threshold>
  Cost:      median TOKEN_CREDITS per request from CORTEX_AGENT_USAGE_HISTORY <= <budget>

EVALUATION DATASET
  Seed: <n> questions (<breakdown by category>), <n> still needs_review
  Target by end of Phase 1: <n>

PHASES
  Phase 1: <scope>  exit: <metric thresholds, pilot users>
  Phase 2: <scope>  exit: <criteria>

ACCESS CONTROL
  Tenant model:        <single / multi-internal / multi-external>
  Identity mechanism:  <Snowflake roles / session attributes>
  Row access policies: <tables, column, logic>
  Masking:             <columns>

KNOWN BLOCKERS
  <prerequisites not yet met>

A phased plan keeps Phase 1 small:

Post-launch: compare real questions with the taxonomy¶

sql/04_observability.sql does three things for a deployed agent:

1. Summarizes requests and credits by user and surface from SNOWFLAKE.ACCOUNT_USAGE.CORTEX_AGENT_USAGE_HISTORY (USER_NAME, TOKEN_CREDITS, TOKENS, METADATA:interaction_interface). Rows can take up to about an hour to appear.

2. Reads the questions users asked from the agent’s monitoring events. Each turn has one record_root span whose ai.observability.record_root.input attribute is the user’s message; evaluation runs write the same spans with a run name, so the script keeps only spans without one:

   Copy code

   SELECT RECORD_ATTRIBUTES:"ai.observability.record_root.input"::VARCHAR AS question
     FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS($agent_db, $agent_schema, $agent_name, 'CORTEX AGENT'))
    WHERE RECORD_ATTRIBUTES:"ai.observability.span_type"::VARCHAR = 'record_root'
      AND RECORD_ATTRIBUTES:"snow.ai.observability.run.name" IS NULL;
   

   The message text is conversation content, so a role sees it only with the READ UNREDACTED AI OBSERVABILITY EVENTS TABLE account privilege. Without it the span is returned with the input attribute removed, so every question is NULL. The script checks for that case and stops with an error instead of reporting 0% real usage for every category. The privilege exposes every conversation the role can monitor, so grant it only to people who may read them.

3. Classifies those questions into the taxonomy with AI_CLASSIFY, using the categories from SEED_QUESTIONS, and shows the share of each category in the evaluation set next to its share of real usage.

A category that is common in real usage but thin in the evaluation set needs more rows. A category that is out of scope but common is a candidate for the next phase.

Verify¶

After sql/03_eval_dataset.sql:

USE ROLE COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS_ROLE;
SELECT GROUND_TRUTH:category::VARCHAR AS category, GROUND_TRUTH:needs_review::BOOLEAN AS needs_review, COUNT(*)
  FROM COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_QUESTIONS GROUP BY ALL ORDER BY 1, 2;
-- With the sample seeds: 3 lookup rows with literal values (needs_review FALSE) and
-- drafted rubrics for the remaining Phase 1 lookup, aggregation, and out-of-scope questions.
SHOW DATASETS IN SCHEMA COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB;
-- SCOPING_EVAL_DATASET

The literal rows read, for example, “Closed-won revenue for the West region between April 1 and June 30, 2024 is $4,200,000.00.”

After uploading the config, SELECT $1 FROM @COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS.LAB.EVAL_CONFIGS/eval_config.yaml returns the file one line per row, which confirms the stage format.

Against a deployed agent, sql/04_observability.sql reports how many real-usage turns it found and how many have readable question text, then ends with a table of category, pct_of_eval_set, and pct_of_real_usage.

Gotchas¶

Teardown¶

Run sql/99_teardown.sql as ACCOUNTADMIN. It drops the lab database (tables, stage, file format, and dataset), the warehouse, and the role. Dropping the role also removes its EXECUTE TASK grant and any MONITOR or READ UNREDACTED AI OBSERVABILITY EVENTS TABLE grant you gave it.

Adapt this recipe¶

Keep these, so a customized version keeps the recipe’s security model:

• Least privilege: ACCOUNTADMIN only for the scripts listed under Privileges; everything else runs as the recipe’s own role.
• No grants to PUBLIC.
• Credentials: create keys, tokens, and secrets yourself, in a separate terminal, and read them from environment variables. Never put them in a script or give them to an agent.
• Dynamic SQL, if you add any, uses bind variables or an allowlist of identifiers, never text concatenated from user or model input.

Change these to fit your use case:

• Object names. Rename them in every script, including teardown, then check that teardown still removes everything.
• Roles, personas, and users: map them to your own, each granted no more than the recipe grants.
• Data: point the scripts at your tables instead of the sample data, then run Verify again.
• Models and warehouse sizes.

To have a coding agent adapt it:

Adapt the scoping-agentic-implementations recipe in scoping-agentic-implementations/ (unzipped from scoping-agentic-implementations.zip) to <describe your use case>.
Read README.md first. Keep everything the "Adapt this recipe" section says to keep.
Change only names, roles and personas, data, and models.
Show me each changed file before you run anything, then run the recipe's Verify steps.

Run it with a coding agent¶

Run the scoping-agentic-implementations lab in scoping-agentic-implementations/.
1. Install or enable the bundled agent-studio and cortex-ai-function-studio skills,
   and the check-agent-access skill from Snowflake-Labs/coco-skills. Use
   cortex-ai-function-studio if the AI_COMPLETE or AI_CLASSIFY calls need tuning,
   agent-studio for evaluation runs, and check-agent-access to check my agent's
   grants before step 5.
2. Confirm the target account with SELECT CURRENT_ACCOUNT(), CURRENT_REGION(), CURRENT_ROLE()
   and show me the result before creating anything.
3. sql/01_setup.sql runs as ACCOUNTADMIN and grants EXECUTE TASK ON ACCOUNT to the lab role.
   Show me its ACCOUNTADMIN statements, wait for my confirmation, then run it with snow sql -f.
4. Replace the seed rows in sql/02_question_taxonomy.sql with these questions for my persona:
   <paste questions>. Keep every date absolute. Run the script and show me ALL_QUESTIONS
   grouped by source and category.
5. Run sql/03_eval_dataset.sql. For each row with needs_review = TRUE, show me the question
   and ground truth so I can approve or rewrite it. Where the answer is a single number in my
   data, compute it with SQL instead.
6. Fill in config/eval_config.yaml for my agent <db.schema.agent> at version <VERSION$N> and
   upload it with snow stage copy ... --role COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS_ROLE.
   Do not start an evaluation until I confirm the dataset.
7. Before sql/04_observability.sql, show me the two GRANT statements in its header. They are
   account-level and expose my users' questions, so run them only after I confirm.
8. When I say the lab is done, run sql/99_teardown.sql and confirm that
   SHOW DATABASES, SHOW WAREHOUSES, and SHOW ROLES LIKE 'COOKBOOK_SCOPING_AGENTIC_IMPLEMENTATIONS%'
   return nothing.

Related skills¶

• agent-studio (bundled): curating an evaluation dataset from production questions, running evaluations, and investigating and optimizing an agent from the results. Use it once the Phase 1 agent exists.
• cortex-ai-function-studio (bundled): authoring and tuning the AI_COMPLETE and AI_CLASSIFY calls used in the lab, including custom AI functions if the built-in classification is not accurate enough for your taxonomy.
• check-agent-access (Snowflake-Labs/coco-skills (https://github.com/Snowflake-Labs/coco-skills)): reads each agent’s live specification, lists every object its tools depend on, and reports which privileges a given role is missing. Use it to check the identity point of The platform on one page against a real agent.

References¶

• Cortex Agents
• Cortex Agent evaluations
• SYSTEM$CREATE_EVALUATION_DATASET
• EXECUTE_AI_EVALUATION
• AI_COMPLETE structured outputs
• Monitor Cortex Agent requests
• Query a Cortex Search Service
• Cortex Agent versioning
• CREATE AGENT
• Snowflake-managed MCP server
• MCP: Streaming responses for tool calls
• MCP connectors
• Per-user quotas
• Workload identity federation