A web service fails loudly: a 500, a timeout, a stack trace. A personal assistant fails quietly — the answer is a little wrong, or a little cold, or slightly less like the person it is talking to than it was last week. At a hundred million weekly users, "it seems fine" is not a test. This chapter is how a lab knows, before, during and after a rollout, that the assistant is still correct, still kind, still fast and still affordable.
Why can't an assistant be tested like a web service?
Three properties of the agentic loop break the classic test pyramid. The model is non-deterministic: the same input yields different words every run, so an exact-match assertion is meaningless. The product is personal: one question has several right answers — the right answer for Maya (short, visual, allergy-aware) is the wrong answer for Raj (numbers, assumptions, sources). And tone is part of correctness: a factually right answer that is dismissive to a sleep-deprived parent is a failed turn.
So the unit of testing changes. Instead of assert equals, quality is a score against a rubric, produced by a judge, aggregated over a fixed set of cases and compared with a threshold. Instead of one suite, there is one per persona. And instead of running once, it runs on every deploy as a gate and continuously in production as a sample. That is what eval means here: a standing instrument, not a benchmark.
🪤 Misconception … "Tone is subjective, so it can't be tested." Readers may disagree at the margins, but tone is not random. Give a calibrated judge a written target — "gentle, non-judgmental, brief" — and it agrees with human raters most of the time and with itself run after run. A regression gate needs only that consistency: not what perfect warmth is, but whether this release is colder than the last.
What is a golden set, and why one per persona?
A golden set is a small, curated list of cases the assistant must keep getting right. Each case names a persona, a question, the behaviour expected (a rubric, never a verbatim answer) and an EQ target — the tone that segment's tone template should produce. Cases are grounded in the persona's seeded memory, so each one tests personalization, retrieval and tone at once: an answer that ignores Maya's allergy fails on substance however charming it is.
JSON
{
"id": "lena-naps",
"persona": "lena",
"question": "I'm worried about Theo's naps. Any thoughts?",
"expected": "Recalls that Theo (18 months) has been refusing naps; offers one simple, non-judgmental suggestion; stays short.",
"eq_target": "Gentle and reassuring; normalizes the worry; brief — she is time-poor and tired."
}
Why per persona? Because averages hide exactly the regressions that matter. A change that makes every answer more thorough will raise Raj's scores and sink Maya's and Lena's; one blended average could stay flat while two of three segments got worse. Per-persona suites, each with its own floor, make that visible. Consumer labs run many more segments — new users, power users, non-English speakers, people in distress — each with its own suite.
How do you score tone as well as facts?
Every answer gets two independent scores from an LLM judge: substance (does it satisfy the rubric — correct, relevant, complete, actionable) and EQ (does it hit the target tone — emotional intelligence, reassurance, register). The judge sees the question, rubric, EQ target and answer, and returns strict machine-readable numbers with a one-line reason. The axes are deliberately decoupled: an answer can be brilliant and cold, or warm and wrong, and the gate needs to see which.
Three disciplines keep the judge honest. It is a different, usually stronger model than the one under test, so it does not share its blind spots. It is calibrated against human-labelled anchor cases whenever the judge model changes. And it is measured like any classifier — its agreement with humans is a number on a dashboard, because a drifting judge silently corrupts every gate downstream.
What is the deploy gate?
flowchart LR
G["Golden set per persona"] --> R["Run against candidate revision"]
R --> J["LLM judge"]
J --> S1["Substance score"]
J --> S2["EQ score"]
S1 --> T{"Both ≥ threshold and no suite below floor?"}
S2 --> T
T -- no --> B["Block deploy report failing cases"]
T -- yes --> C["Canary small slice of traffic"]
C --> W{"SLOs hold? online sample holds?"}
W -- no --> RB["Roll back"]
W -- yes --> P["Promote to all traffic"]
The gate runs on every candidate — a new prompt, tone template, retrieval setting or model version — not on a schedule. The candidate answers every golden case, the judge scores each, and the run passes only if both average scores clear the threshold and no persona suite falls below its floor. A failing run blocks the rollout and lists the cases that regressed.
Passing earns a canary, not a promotion. A small slice of real traffic goes to the new revision while its latency, error rate and a sampled online eval are watched; only when those hold does the split ramp (mechanics in Model Serving). The two stages matter because the golden set is a fixture — it cannot contain the question a real person will ask in ten minutes. Offline eval catches regressions on what is known; the canary catches the rest.
⚠️ Pitfall … gating on one blended average. A release that lifts the overall score from 0.81 to 0.83 while dropping one persona's EQ from 0.85 to 0.60 passes a naive gate and ships a colder assistant to a whole segment. Set floors per suite and per axis.
What do SLOs promise, and what does an error budget buy?
An SLO is a promise expressed as a number over a window. For a conversational assistant the promises that matter are latency to the first streamed token (people perceive the first word, not the last), availability, and the rate of turns that end in an error or a fallback. Each objective implies an error budget — the failure permitted before the promise is broken — and that budget is the most useful operational number the system produces.
Objective
Target (example)
Window
Error budget it implies
Latency to first token, p50
≤ 900 ms
30 days
tracked; informs capacity, not a hard promise
Latency to first token, p95
≤ 2.5 s
30 days
5 % of turns may exceed
Availability (turn answered)
99.9 %
30 days
~43 minutes of downtime, or 1 turn in 1 000
Error rate (turn failed or degraded)
≤ 0.5 %
7 days
1 turn in 200
Cost per turn, p95
≤ 1.5× the p50
7 days
catches runaway tool loops and bloated packs
The budget governs rollout risk directly. With most of the month's budget unspent, canaries ramp quickly and several changes ship in a day. When the budget is burning — a bad model day, a retrieval store under load — policy tightens automatically: canaries hold at their smallest slice, non-urgent changes queue, and only deploys meant to stop the burn proceed. Nobody argues about whether now is a good time; the budget already answered.
stateDiagram-v2
[*] --> Healthy: most of the budget left
Healthy --> Cautious: burn rate rising
Cautious --> Frozen: budget nearly spent
Frozen --> Cautious: burn stops
Cautious --> Healthy: window rolls over
Healthy: canaries ramp fast, many deploys a day
Cautious: canary holds at its smallest slice, one deploy at a time
Frozen: fixes only, no feature rollouts
🎯 At consumer scale … the SLO becomes a matrix — per region, per model tier, per client platform — because a p95 that is fine globally can be broken for everyone in one region routed to one overloaded tier. The reconcile loop in the control plane consumes these signals too: queue depth and latency are scaling triggers, so an SLO breach can cause a scale-out rather than merely report one.
Where does the money go on one turn?
Because the bill is tokens, cost is a quality metric, accounted per turn, per stage and per person. Every model call records its input and output token counts and the tier that served it; the turn total is the sum, and the running total per user per day is what product and finance look at.
Stage
Model calls
Typical tokens (in / out)
Share of turn cost
Input guardrail
0
0
0
Query rewrite
1 (small tier)
300 / 30
~3 %
Retrieval (profile + memory)
0 (embedding only)
40
under 1 %
Context pack assembly
0
budgeted to ~2 000
—
Agent + tools
1–3 (routed)
2 200 / 150 per call
~40–60 %
Generate + output guardrail
1 (routed)
2 400 / 250
~35–50 %
Write-back (async)
1 (small tier)
400 / 60
~3 %
Two things fall out. The context pack dominates input tokens on every priced call — which is why the pack has a hard budget and why prefix caching of the stable prompt prefix (tone template, tool catalogue) pays for itself at once. And the agent stage is the only line with a multiplier — each extra iteration of tool use re-sends the pack — which is why tool loops carry caps and cost per turn has its own SLO. A cost p95 climbing while p50 stays flat is the signature of a runaway loop, long before the invoice.
How do you answer "what happened on that turn"?
Observability for an assistant means reconstructing one turn after the fact. Every turn gets a turn id minted at the edge, and every stage appends to a record keyed by it: guardrail flags, the session loaded, the rewritten query, the profile keys and memory chunk ids retrieved, what the pack budget trimmed, each tool call and outcome, the model and tier, output flags, token counts and latency per stage. Metrics, logs and traces all carry the same id, so a complaint, an SLO alert and a cost spike each walk back to the exact turns involved.
The record stores identifiers and counts, not contents: chunk ids rather than memory text, a PII flag rather than the PII, never a secret. That is what makes it safe to keep, to sample into eval and to show an operator — and what Trust relies on when it insists nothing sensitive is ever logged.
📖 Story: the release that was correct and cold
A prompt change meant to make answers "more thorough" ships on a Friday. Substance scores rise; latency is fine. Over the weekend the people who write to the assistant when stressed — new parents, anyone up at 3 a.m. — quietly stop coming back, because replies got longer and the warmth got buried under bullet points. No alert fires; nothing is broken. The team learns from retention two weeks later. The fix is not a better prompt: it is a second score, a floor per segment, and a gate that would have blocked the change before it reached anyone.
🔍 See it happen — a regression caught before the canary
A change to the "relational" tone template lands: a sentence asking the assistant to "give complete context".
The gate runs the three persona suites. Raj: EQ 0.84 → 0.85. Maya: EQ 0.83 → 0.71. Lena: EQ 0.87 → 0.62. Substance flat or up everywhere.
The blended averages clear the 0.70 threshold; Lena's suite misses its 0.75 EQ floor. Run marked blocked.
The report lists lena-naps with the judge's reason: "accurate but long and instructional; does not normalize the worry."
The author shortens the change and re-runs. Lena's EQ returns to 0.85. The candidate goes to a 2 % canary; first-token p95 and fallback rate hold for an hour; the split ramps to full traffic, every canary turn tagged with its revision in the turn record.
🧭 Enterprise mapping … the golden set becomes compliance evidence: a bank's advisor assistant keeps a suite of regulated scenarios (suitability, disclosure, "I can't advise on that") and the gate report is what an auditor asks for. The EQ axis becomes a policy axis — brand voice, mandated disclaimers, escalation language — scored the same way. SLOs become contractual SLAs with penalties, so the error budget is money. And the turn record becomes the audit trail regulators expect — who asked, what was retrieved, what was said, which model, at what cost, retained for years — under the same rule: identifiers and flags, never secrets or raw personal data.
Next: Trust — what the guardrails in and out actually check, why the memory namespace is the privacy boundary, and how the assistant degrades gracefully when the model is not there.