One Turn — what happens in the 900 ms after you press send

A person on a phone types "any good trails near me this weekend?" and presses send. Nine hundred milliseconds later a reply begins to appear, addressed to her — her city, her Saturday habit, her taste for short answers — from a worker that had never heard of her a moment earlier. This chapter is the 900 ms in between: nine stages that turn a bare message into a personal answer and then learn from it. Personal is not a bigger prompt; it is a per-turn context pack, and every stage below feeds that pack, spends it, or replenishes it.

What are the nine stages?

1. Guardrail in — rules before any token is spent.
2. Session — recent turns, from a shared store.
3. Query rewrite — make "there this weekend" standalone.
4. Profile retrieval — keyed lookup ⊕ vector search in the person's memory namespace.
5. General retrieval — shared knowledge that belongs to nobody.
6. Context pack — profile ⊕ tone template ⊕ memory, under an allocation budget.
7. Agent + tools — reason, call, observe, repeat.
8. Generate (+ output guardrail) — stream through a leak-and-grounding check.
9. Write-back — store what was learned, off the hot path.

sequenceDiagram
    autonumber
    participant P as Person
    participant W as Stateless worker
    participant S as Session store
    participant M as Profile + memory
    participant L as Model
    participant T as Tools
    P->>W: message
    W->>W: 1 guardrail in (rules, no tokens)
    W->>S: 2 load session (wait on write-back lock)
    W->>L: 3 query rewrite (small model)
    par
        W->>M: 4 keyed profile + top-K memory (own namespace)
    and
        W->>M: 5 shared knowledge (no user filter)
    end
    W->>W: 6 context pack under budget
    loop 7 agent + tools — until done or a cap trips
        W->>L: reason over the pack
        L-->>W: tool call?
        W->>T: call tool (user id injected)
        T-->>W: result (or error as data)
    end
    W->>L: 8 generate
    L-->>W: tokens stream
    W->>W: output guardrail (leak + grounding)
    W-->>P: answer, streamed
    W--)M: 9 write-back (async, per-user lock)
    W--)S: refresh session

Why does a guardrail run before any model call?

The first thing that touches the message is not a model but microseconds of rules: a length bound, a few prompt-injection patterns, a scan for personal identifiers. It runs first because a rejected message should cost zero tokens, and because the client is untrusted: this one string will flow into a rewrite prompt, an embedding, tool arguments and several model calls, every hop an injection surface.

The design choice is flag, don't block. Only an over-length message is refused; suspected injection and identifiers become flags that ride with the turn so later stages, and the output guardrail at stage 8, stay defensive.

🪤 Misconception — "the guardrail is the security." It is one layer. Input heuristics make risk visible; the output guardrail catches leakage; the tool layer never lets the model choose whose data a tool reads. See Trust.

How does a stateless worker know what "there" means?

The worker holding this turn has no memory of the last one, so stage 2 reads the session from a store every worker shares. One race matters: if the previous turn is still writing back what it learned, this turn could read a stale snapshot. A short per-user lock fixes it — write-back holds it, the session load waits on it.

Stage 3 is the first model call. "What about there this weekend?" embedded as-is lands in the wrong region of vector space, so a query rewrite uses recent history to produce "hiking trails at Mount Tam this weekend", and that rewritten query — never the raw message — is what stages 4 and 5 search on. A small model does it; the raw message is the fallback.

Where does the personal part come from?

Stage 4 performs two very different retrievals. A keyed lookup fetches the profile card by primary key: exact, always relevant, never left to similarity. Then vector search runs over the person's episodic memory, filtered to their memory namespace before anything is ranked — that filter is the privacy boundary of the whole system — and re-scored with temporal decay, so last week's preference outranks a contradictory one from three years ago.

Stage 5 searches a separate, shared corpus with no user filter, kept apart so one person's memories can never surface in another's results and because similarity scores from different corpora are not comparable. Stages 4 and 5 run in parallel.

Why is assembling the prompt a budgeting decision?

Stage 6 builds the context pack — assembly under a constraint, not concatenation. The pack has an allocation budget: fixed slots for personal memory and for general knowledge (three and two is a sound default), an authoritative block for the keyed profile, and the tone template chosen by the person's segment. The budget is cost control and personality control at once: fewer, better chunks beat more, noisier ones, and attention is finite even when the context window is not.

Where is the "agentic" in the agentic loop?

In a conventional service the engineer is the router. In stage 7 the model is the router. It sees the pack plus a self-describing catalog of tools — discovered through MCP, not hard-coded — and decides whether to call one; the worker executes the call, appends the result, and asks again. Plan, act, observe, repeat.

Because the model decides at runtime, the loop needs caps it cannot talk its way past: a maximum number of iterations, a cost and latency budget, detection of the same call failing twice, and tool-error isolation — a failed tool returns its error as data the model can reason about, never an exception that kills the turn. For memory-reading tools the worker injects the current person's identity; the model never chooses whose memory to read. Tools covers tool use.

What checks the answer on the way out?

Stage 8 produces the answer and passes it through an output guardrail, the mirror of stage 1. A leakage scan refuses anything shaped like a system-prompt header or a credential. A grounding check verifies that every citation marker the answer uses exists in the pack's registry — a citation to a source never retrieved is a hallucination wearing a footnote. Streaming changes nothing: the guard runs on chunks.

Why does learning happen after the reply?

Stage 9 makes the assistant compound. A small model extracts a few durable facts from the turn — "prefers hiking", "lives near Austin" — each checked against existing memory by exact text and by vector near-duplicate, so saying "I love hiking" five times yields one memory, not five. New facts are stored in the person's namespace; hot preferences are mirrored into the session. None of this sits on the person's latency path: the reply has already streamed.

Which stages are synchronous, and what does each cost?

The 900 ms headline is a tool-free turn measured to first visible token. Each tool round in stage 7 adds another model round-trip plus the tool's latency. The bill is almost entirely stage 7 — tokens in the pack multiplied by iterations — which is why budgeting the pack and capping the loop are the levers that matter.

flowchart TD
    A[Message arrives] --> B{Over length?}
    B -- yes --> C[Canned refusal — zero tokens]
    B -- no --> D[Stages 2–6: assemble the pack]
    D --> E{Model wants a tool?}
    E -- no --> H[Final answer]
    E -- yes --> F{Cap tripped?
iterations · cost · time · repeat-fail}
    F -- yes --> H
    F -- no --> G[Call tool, feed result back] --> E
    H --> I{Leak or ungrounded citation?}
    I -- yes --> J[Redact / flag]
    I -- no --> K[Stream to the person]
    J --> K
    K -.-> L[Write-back, off the hot path]

🎯 At consumer scale, this ordering keeps the worker stateless and the cost per turn flat. Because no turn depends on the previous one's machine, a lab serving 100M weekly users treats every turn as an independent unit of work and sizes the fleet on aggregate load. The expensive call at stage 7 is the one that gets routed and tiered — see Model Serving.

⚠️ Pitfall — running write-back inline. It adds a model call and a store write to every reply the person is waiting for, and invites the stale-read race unless the lock exists. Keep learning asynchronous; keep the lock.

🧭 Enterprise mapping. The nine stages survive the move to a bank's advisor assistant or a retailer's support agent unchanged; what changes is the strictness at each gate. Session and memory become tenant-scoped, region-pinned stores with retention rules. Stage-7 tool calls acquire per-tenant allow-lists and, for consequential actions, a human-in-the-loop step. The output guardrail adds a citation-required rule for regulated advice. Write-back becomes an auditable event keyed by turn id. See Consumer → Enterprise.

Where next?

Personalization unpacks the stages that make a turn personal; Memory shows how it compounds; Tools opens the loop inside stage 7; The Fleet explains the stateless worker and the control plane that sizes the fleet. Every term above has a card in the Glossary.