Model Serving — routing, tiering, and why the bill is tokens

Earlier chapters treated "the model" as a box the turn passes through. This chapter opens the box from the outside: how a lab decides which model answers a turn, what it costs, what happens when a provider fails, and how a new model reaches a hundred million people safely. The thesis, said the serving way: scale is not a bigger model on a bigger box — it is a routing policy that spends frontier-model tokens only where they change the answer.

Why is the model a service, not a library?

A stateless worker does not own a GPU. It assembles the context pack, then calls a model service over the network — an endpoint that accepts a prompt and returns (or streams) tokens. That separation is why the fleet's reconcile loop can scale workers on active users while model capacity scales on a different signal: tokens per second, queue depth at the accelerator, GPU memory.

The separation also makes the model replaceable per turn. Behind one interface a lab keeps a frontier tier for hard or sensitive turns, a small/fast tier for easy ones, and a second provider as a fallback. That interface, plus a policy for choosing, is model routing.

🪤 Misconception — "one great model is simpler than a router." For a week. At scale a single tier means every "thanks" pays frontier prices and every provider incident is a full outage.

Why is the bill tokens, not servers?

In a web service the cost driver is compute-seconds, and "hello" costs about the same as a paragraph. A model call costs in proportion to the tokens it reads and writes. The context pack from the personalization chapter — profile facts, a tone template, top-K memories, recent history — is read on every turn, so its size is a recurring charge × turns per day × users. Output tokens are priced several times higher than input, so verbosity is also a bill.

Three levers follow:

At planet scale the fleet is a rounding error; model tokens dominate the ledger. Past a certain size, "how do you scale?" is answered by "how do you route?"

How does a turn pick its model?

The router runs once per turn before the agentic loop begins (and may run again inside it, since a tool-heavy iteration and a final wording pass want different models). It weighs intent, cost budget and latency budget, chooses a tier, and attaches a fallback chain.

flowchart TD
  T[Turn + context pack] --> C{Intent class?}
  C -->|chit-chat, short follow-up| S[Small / fast tier]
  C -->|reasoning, tools, sensitive| F[Frontier tier]
  C -->|unsure| L{Latency + cost
budget left?}
  L -->|yes| F
  L -->|no| S
  F -. on error / timeout .-> F2[Second provider]
  F2 -. on error .-> S
  S -. on error .-> P[Answer from the pack
no model]

Intent comes from the query rewrite stage: a cheap classifier (or the small tier itself) labels the turn. Cost budget is per-turn and per-user-per-day — a heavy user who has spent the day's allowance is routed down a tier, not refused. Latency budget is what remains of the turn's target after retrieval; a slow pack build must not also buy the slowest model.

The fallback chain is part of the decision. Each hop degrades gracefully: frontier model → a second provider of similar capability → the small tier → an answer composed from the pack alone (a keyed fact plus the tone template), so the user never sees a raw error.

⚠️ Pitfall — routing on the question alone. "What's a good dinner idea?" is an easy turn — unless the profile says the person is grieving. Routing must see the segment and safety flags, not just the text, or the cheap tier answers exactly the turns where empathy mattered most.

What does tiering actually save?

The table below is deliberately simple so the shape of the curve is visible. Every number is an assumption to replace with your own measurements; the prices are illustrative placeholders, not any vendor's list price.

Assumptions. 3,000 input tokens per turn (pack + history + instructions), 300 output tokens, 20 turns per daily-active user. Illustrative prices: frontier tier $3 / 1M input, $15 / 1M output; small tier $0.10 / 1M input, $0.40 / 1M output. "Blended" routes 70% of turns to the small tier; "Blended + cache" also serves 60% of input tokens from a prefix cache at one-tenth of the input price.

Read the last column. At a hundred million daily users, "all frontier" is roughly ten billion dollars a year; the blended-and-cached policy is about a fifth of that, with the same frontier model on the turns that need it. A routing policy did that, not a fleet change. At 1K users the difference is a restaurant bill — which is why small teams correctly ignore routing until the curve turns.

🎯 At consumer scale, the router is the most valuable eval subject in the company. Moving five percent of turns down a tier is worth more than most infrastructure projects — and is only safe if a golden set proves those turns did not get worse.

What happens when a provider fails?

Providers have incidents, rate-limit under load, and sometimes return slowly rather than failing. A serving layer treats each tier as an unreliable dependency:

• Timeouts per tier, shorter than the turn's latency budget, so a slow provider triggers the fallback instead of consuming the whole budget.
• Circuit breaking: after a burst of errors the router stops trying that tier for a cooling period and sends traffic straight to the next hop, protecting the SLO and the error budget.
• Capability-aware fallback: the next hop must support what the turn needs (tool use, a long context window, streaming); otherwise fall to the pack-only answer, not a model that silently drops the tools.

Standard upstream discipline — with one twist: circuit breaking and fallback may answer differently, so the trace must record which tier actually served the turn. The quality chapter depends on that field.

How do caching and streaming cut latency and cost?

Prefix caching. Much of every prompt is identical across turns: the system instructions, the tone template for a segment, and — within one session — the earlier conversation. Model services can cache the computed state of a prompt prefix and charge a fraction to reuse it. The design consequence is ordering: stable parts first (instructions, template), per-user parts next (profile, memories), volatile parts last (the new message). A pack assembled in that order caches well; one that interleaves a timestamp into the system prompt never does.

sequenceDiagram
  participant W as Worker
  participant M as Model service
  W->>M: [cached prefix: instructions + tone template] + [profile + memories] + [new message]
  M-->>M: reuse prefix state, compute only the tail
  M-->>W: first token in ~200 ms
  M-->>W: stream remaining tokens

Streaming. A three-second answer feels fast if the first word appears in a few hundred milliseconds. Streaming does not reduce cost, but it changes the latency metric that matters from "time to complete" to time to first token — and it moves the output guardrail onto the stream, running incrementally with a hold-back buffer rather than once on the finished text.

How does a new model reach users safely?

A new model version is a deploy, and it gets a canary like any other. The router holds a traffic split: most turns to the stable model, a small slice to the candidate. Both slices are scored the same way — online signals (latency, refusals, thumbs-down, tool-error rate) and the offline golden set with its two scores, correctness and empathy. The candidate is promoted only when the eval gate passes.

flowchart LR
  R[Router] -->|95%| A[Stable model]
  R -->|5%| B[Candidate model]
  A --> E[Eval: online signals + golden set]
  B --> E
  E -->|gate passes| P[Widen split 5 → 25 → 100%]
  E -->|regression| K[Roll back to 100% stable]

The split is by user, not by turn, so one person never gets two personalities in one conversation. And the canary is per tier: a new small model can roll out while the frontier tier stays pinned.

🧭 Enterprise mapping — the router is where an enterprise expresses policy: this tenant's data may only go to a model hosted in-region; that tenant bought a frontier-tier SLA; a regulated workflow needs a signed eval report before any model change. The mechanism is unchanged — tiers, fallback chains, a traffic split gated by eval — but the routing table becomes a contract, the cost table a per-tenant invoice, and "fallback to another provider" may be forbidden by a residency clause. See the enterprise chapter.

Where to next

Routing chooses the model; the next chapter is about what that model can do mid-turn: Tools — letting the model act, safely. The quality chapter explains the eval gate this chapter leaned on; the fleet chapter explains the workers calling the model service.