The golden set is a product asset

Testing Copilot Studio and AI agents

Last Episode we said the golden set anchors everything else: regression, judge calibration, release evidence and incident memory. This Episode we build one for Mira, the Meridian Bank agent. We cover what goes in it, how big it should be, how it’s versioned and reviewed, and how to let AI help write it without letting AI decide what “correct” means.

The main point is simple. A golden set is a product asset, not a test artefact. It has an owner, a schema, a review process and a release history, the same as the agent. If nobody owns it, it rots, and a rotten golden set is worse than none because it produces confident green ticks.

Start with the standard

A golden case only means something when it is checked against an explicit definition of “good”. We use the six dimensions from Episode 1:

Six dimensions of good

Not every case tests every dimension. A product-FAQ case tests correct, grounded and useful. A card-freeze case tests action-accurate and safe. A “show me my colleague’s balance” case tests safe and nothing else. Writing down which dimensions a case covers is what lets you choose the right grader later (Episode 3) and spot coverage gaps.

What goes in: four kinds of case

Golden set composition

Normal behaviour (about 40% at first release). These are the tasks customers rely on every day: balance, recent transactions, how to dispute, branch booking, ISA rates. They prove the agent does its job.

Edge conditions (about 25%). These are ambiguous, incomplete, multi-intent and out-of-scope requests. “Move it to savings” (which account? how much?). “Freeze it and send me a new one”. “What’s the rate on the Flexi-Saver Plus?” (a product that doesn’t exist). Most production failures live here.

Adversarial (about 20%). Direct and indirect injection, role-play jailbreaks, and requests for another customer’s data. Episode 7 expands these a lot. The golden set holds a curated core of the cases that must always fail safely.

Known regressions (about 15%, and growing). Every past failure that must never happen again. This category only grows. After a year in production it is often the most valuable part of the set.

The mix moves with risk. A low-risk internal FAQ agent might be 60/25/10/5. Mira, which moves money for external customers, ends up much heavier on adversarial and regression cases.

How many cases?

Use these as starting points, not targets:

Agent risk tier	Starting size	Mature size	Repeats per critical case
Low	30–50	50–100	1–3
Moderate	50–100	100–200	3–5
High	100–150	200–400	5–10
Critical	150+	400+	10+

Two rules matter more than the totals:

Stratify. Every high-impact action, every knowledge source and every persona needs its own cases. A set of 300 cases with nothing on InitiateTransfer has zero coverage of the riskiest thing Mira does.
Repeat the critical cases. One pass on a probabilistic system is one sample, not a proof. Episode 4 shows how many repeats you need before a pass rate means anything.

One practical limit: a single-response test set in Copilot Studio holds up to 100 test cases. Imports are CSV or TXT files with Question and Expected response columns, and each question can be up to 1,000 characters (Microsoft Learn[1]). Keep one canonical golden set in git and export it into several themed Copilot Studio test sets, for example “Cards”, “Payments”, “Product FAQ” and “Adversarial core”.

Anatomy of a golden case

Anatomy of a golden case

The question is the least interesting field. The fields that make a case useful are the ones a transcript can’t give you:

identity: which test persona asks. This decides what Mira is *allowed* to see and do.
context: earlier turns and variables that must carry through. Episode 3 covers conversation test sets.
expected sources: which approved document or record the answer must rest on.
expected route: which tool, with which arguments, in which order, and whether confirmation is required.
prohibited: actions or disclosures that fail the case outright, however good the wording.
acceptance: a plain-English criterion that a human and a calibrated judge can both apply.
severity and owner: how the gate treats a failure, and who fixes it.

The schema, as code

We validate every case on commit. This is the Pydantic model we use:

code/golden-set/golden_schema.py from __future__ import annotations from enum import Enum from typing import Optional from pydantic import BaseModel, Field, field_validator class Category(str, Enum): normal = “normal”; edge = “edge”; adversarial = “adversarial”; regression = “regression” class Severity(str, Enum): critical = “critical”; high = “high”; medium = “medium”; low = “low” class ExpectedCall(BaseModel): tool: str args: dict[str, str] = Field(default_factory=dict) requires_confirmation: bool = False max_calls: int = 1 class Expected(BaseModel): route: list[ExpectedCall] = Field(default_factory=list) # empty list = no tool may fire sources: list[str] = Field(default_factory=list) # approved doc IDs / record types must_mention: list[str] = Field(default_factory=list) must_not_mention: list[str] = Field(default_factory=list) reference_answer: Optional[str] = None # for compare-meaning graders outcome: str = “answer” # answer | fallback | refuse | handoff class Turn(BaseModel): user: str expect_contains: list[str] = Field(default_factory=list) critical: bool = True class GoldenCase(BaseModel): id: str version: int category: Category severity: Severity owner: str dimensions: list[str] = Field(default_factory=lambda: [“correct”, “grounded”]) identity: dict[str, str] input: Optional[str] = None # single-turn case turns: list[Turn] = Field(default_factory=list) # multi-turn case paraphrases: list[str] = Field(default_factory=list) expected: Expected prohibited: list[str | dict] = Field(default_factory=list) acceptance: str source_incident: Optional[str] = None # set for regression cases @field_validator(“id”) @classmethod def id_format(cls, v: str) -> str: if not v.startswith(“GS-“): raise ValueError(“ids start with GS-“) return v def model_post_init(self, _ctx) -> None: if not self.input and not self.turns: raise ValueError(f”{self.id}: needs input or turns”) if self.category == Category.regression and not self.source_incident: raise ValueError(f”{self.id}: regression cases must link the incident they came from”) if self.acceptance.startswith(“REVIEW:”) and not self.id.startswith(“GS-CAND-“): raise ValueError(f”{self.id}: promoted case still has an unreviewed acceptance criterion”) if len(self.input or “”) > 1000: raise ValueError(f”{self.id}: Copilot Studio caps questions at 1,000 characters”)

Validate, measure coverage, export

A small CLI runs in pre-commit and CI. It fails the build if a case is malformed, if coverage falls below the floor for any tool or source, or if an expected outcome changed without a reviewer.

code/golden-set/golden_cli.py “””python golden_cli.py validate|coverage|export –set golden/ [–out exports/]””” import csv, sys, pathlib, collections, yaml, argparse from golden_schema import GoldenCase TOOLS = [“LookupBalance”, “FreezeCard”, “InitiateTransfer”, “SubmitDispute”, “BookBranch”, “LoanPreQual”] SOURCES = [“ISA-Terms-2026”, “Fees-2026”, “Disputes-Policy-v7”, “Complaints-FOS-v3”] MIN_PER_TOOL, MIN_PER_SOURCE = 5, 5 def load(path: str) -> list[GoldenCase]: cases = [] for f in sorted(pathlib.Path(path).rglob(“*.yaml”)): cases.append(GoldenCase(**yaml.safe_load(f.read_text(encoding=”utf-8″)))) ids = [c.id for c in cases] dupes = [i for i, n in collections.Counter(ids).items() if n > 1] if dupes: sys.exit(f”duplicate ids: {dupes}”) return cases def coverage(cases: list[GoldenCase]) -> int: by_tool = collections.Counter(call.tool for c in cases for call in c.expected.route) by_src = collections.Counter(s for c in cases for s in c.expected.sources) by_cat = collections.Counter(c.category.value for c in cases) gaps = [f”tool {t}: {by_tool[t]}” for t in TOOLS if by_tool[t] < MIN_PER_TOOL] gaps += [f”source {s}: {by_src[s]}” for s in SOURCES if by_src[s] < MIN_PER_SOURCE] print(“category mix:”, dict(by_cat)) for g in gaps: print(“COVERAGE GAP”, g) return 1 if gaps else 0 def export(cases: list[GoldenCase], out: str) -> None: “””One Copilot Studio test-set CSV per owner, max 100 rows, headings Question / Expected response.””” pathlib.Path(out).mkdir(exist_ok=True) groups = collections.defaultdict(list) for c in cases: if c.input: # multi-turn cases go to conversation sets instead groups[c.owner].append(c) groups[c.owner] += [c.model_copy(update={“input”: p}) for p in c.paraphrases] for owner, rows in groups.items(): for n in range(0, len(rows), 100): path = pathlib.Path(out) / f”{owner}-{n // 100 + 1}.csv” with path.open(“w”, newline=””, encoding=”utf-8″) as fh: w = csv.writer(fh) w.writerow([“Question”, “Expected response”]) for c in rows[n:n + 100]: w.writerow([c.input, c.expected.reference_answer or c.acceptance]) print(“wrote”, path) if __name__ == “__main__”: ap = argparse.ArgumentParser() ap.add_argument(“cmd”, choices=[“validate”, “coverage”, “export”]) ap.add_argument(“–set”, default=”golden”) ap.add_argument(“–out”, default=”exports”) a = ap.parse_args() cs = load(a.set) print(f”{len(cs)} cases valid”) if a.cmd == “coverage”: sys.exit(coverage(cs)) if a.cmd == “export”: export(cs, a.out)

Notice what the export does with paraphrases. Each golden case expands into several test rows. Drift often shows up first on a phrasing nobody used in the original demo, and exact phrasing alone can hide it.

Versioning and review: the golden set has a release history

Golden set lifecycle

We treat the golden set like code:

It lives in the same repo as the agent’s solution, under tests/golden/, so a pull request that changes a topic and the cases it affects goes through review together.
CODEOWNERS protect expected outcomes. Anyone can add a case. Changing an expected or acceptance field needs approval from the case owner and, for regulated wording, from compliance.

code/golden-set/CODEOWNERS /tests/golden/**/GS-FOS-*.yaml @meridian/compliance-reviewers @meridian/cards-squad /tests/golden/**/GS-PAY-*.yaml @meridian/payments-squad @meridian/appsec /tests/golden/** @meridian/agent-factory-leads

Re-baselining is a reviewed change, never a silent edit. The worst anti-pattern in agent testing is editing the expected answer so a red test goes green. If Mira’s new answer really is better, the PR says why, links the evidence and gets approved. If it isn’t, you have found a regression.
Version every case. The version field lets you trend results across releases even after a case’s wording improves.

Synthetic data only

Golden cases are shared widely: in repos, pipelines and test logs, and with the vendors whose tools you use for scanning. Never put real customer data in them. Mira’s personas are synthetic Dataverse test users with synthetic accounts. Regression cases that come from real incidents are sanitised before they’re committed (Episode 10 shows the workflow).

AI in the loop: generating cases without outsourcing judgement

This is where AI has changed testing the most, and where it needs the most care.

What AI is good at: volume and variety. Copilot Studio can generate test cases from your agent’s knowledge sources (it scans them to create relevant questions) or from its topics. It can also seed sets from analytics themes or from your latest test chat (Microsoft Learn[2]). Outside the studio, a model given your tool schemas will produce dozens of edge and multi-intent phrasings in seconds.

What AI is bad at: knowing what your organisation considers correct. A generated question about ISA withdrawal rules comes with a generated expected answer. That answer comes from the same documents the agent uses. If those documents are wrong, stale or poisoned, the test agrees with the bug.

So we use a candidate, review, promote pipeline:

code/golden-set/generate_candidates.py “””Generate *candidate* edge and multi-intent cases for a tool. Output lands in golden/candidates/ and is never picked up by the gate until a human moves it into golden/ via PR.””” import json, os, pathlib, yaml from openai import AzureOpenAI # pip install openai client = AzureOpenAI(azure_endpoint=os.environ[“AOAI_ENDPOINT”], api_key=os.environ[“AOAI_KEY”], api_version=”2024-10-21″) PROMPT = “””You write test cases for a retail-bank customer-service agent. Tool under test: {tool} — {description} Write {n} realistic customer utterances that should trigger the tool AND {n} that are adjacent but must NOT trigger it (vague worries, other intents, two intents in one message). Return JSON: {{“should_trigger”: […], “must_not_trigger”: […]}}. No real names or account numbers.””” def generate(tool: str, description: str, n: int = 8) -> dict: r = client.chat.completions.create( model=os.environ[“AOAI_DEPLOYMENT”], response_format={“type”: “json_object”}, temperature=0.9, # we want variety here messages=[{“role”: “user”, “content”: PROMPT.format(tool=tool, description=description, n=n)}]) return json.loads(r.choices[0].message.content) if __name__ == “__main__”: out = pathlib.Path(“golden/candidates”); out.mkdir(parents=True, exist_ok=True) tool, desc = “FreezeCard”, “Temporarily freezes a debit or credit card the caller owns.” data = generate(tool, desc) for i, u in enumerate(data[“must_not_trigger”], 1): case = {“id”: f”GS-CAND-{tool}-{i:02d}”, “version”: 1, “category”: “edge”, “severity”: “high”, “owner”: “cards-squad”, “identity”: {“persona”: “retail_customer_aosei”}, “input”: u, “expected”: {“route”: [], “outcome”: “answer”}, “acceptance”: “REVIEW: confirm FreezeCard must not fire and describe the correct response.”, “prohibited”: [{“tool”: tool}]} (out / f”{case[‘id’]}.yaml”).write_text(yaml.safe_dump(case, sort_keys=False), encoding=”utf-8″) print(f”wrote {len(data[‘must_not_trigger’])} candidates for human review”)

Three rules keep this honest:

Generated cases start as candidates. They live in a folder the gate ignores until a human promotes them.
A human writes or approves the expected outcome. The acceptance field starts with REVIEW: on purpose, and the validator rejects any promoted case that still contains it.
Generated volume never replaces curated depth. Our working ratio for Mira is about one curated critical case for every five generated variants.

Mira’s case file: the “current” collision

Before. A maker edited the “Report a lost card” topic and added the trigger phrase “my current card”. The next day, “What’s the current Fixed-Rate ISA rate?” started returning the lost-card fraud-hotline script, because both topics now matched on “current”. Nobody had tested the ISA question after a card-topic edit.

The golden-set response. After the incident was fixed, a sanitised regression case went into the set:

code/golden-set/examples/GS-REG-0042.yaml id: GS-REG-0042 version: 1 category: regression severity: high owner: savings-squad source_incident: INC-2026-0815 dimensions: [correct, grounded, action-accurate] identity: { persona: retail_customer_generic } input: “What’s the current Fixed-Rate ISA rate?” paraphrases: – “current rate on the fixed ISA?” – “whats your fixed rate isa paying at the moment” expected: route: [] # no tool may fire; this is a knowledge answer sources: [ISA-Terms-2026] must_mention: [“Fixed-Rate ISA”] must_not_mention: [“fraud”, “lost card”, “0800”] outcome: answer acceptance: > States the Fixed-Rate ISA rate from ISA-Terms-2026 with a citation. Never routes to the lost-card or fraud flow.

After. The case runs on every publish. When a different maker added “current account” as a trigger to the transfers topic three months later, the build went red before the change reached a customer.

Try this week

Write 10 golden cases for your agent: 4 normal, 3 edge, 2 adversarial and 1 regression from a real past issue.
For each, fill in identity, expected route and prohibited. If you can’t fill them in, you’ve found a gap in your agent’s design, not just in its tests.
Run the coverage command against your tool and source list. Any tool with fewer than five cases is untested.

The golden set is a product asset

Start with the standard