Reference Architecture — the build that proved it

Every other chapter explains a concept. This one describes the reference build those concepts were tested on: a Pi-style personal assistant for three fictional people (Maya, Raj, Lena), with a hand-built control plane you can watch, deployed for real on an edge platform plus one cloud region. It is described in a neutral voice and at the altitude of components and choices — not a tour of screens, not a code listing. The point of a reference build is that the ideas on this site were not asserted; they were run, measured and, in one case, honestly missed.

One sentence version of the thesis as this build states it: personalization is a context pack, scale is stateless workers plus a reconcile loop, and the same loop a managed autoscaler runs in production is the one drawn by hand on the dashboard.

What was built, in one picture?

The topology has five lanes. Requests enter at the edge, are served by autoscaled stateless compute, read and write shared state, defer slow work to a queue, and are watched and gated by cross-cutting services.

flowchart TB
  subgraph EDGE["Edge / UI (global CDN)"]
    UI["Chat UI + inspector"] --> GATE["Gate + proxy (edge)"]
  end
  subgraph COMPUTE["Autoscaling compute (managed autoscaler)"]
    API["API gateway"] --> WK["Agent workers · the fleet"]
    EV["Eval / controller service"]
  end
  subgraph STATE["State"]
    DB["Relational store + vector index"]
    HOT["Hot store · sessions, rate limits, fleet state"]
    VS["Managed vector search (at scale)"]
  end
  subgraph ASYNC["Async"]
    Q["Queue · write-back, eval runs"]
  end
  subgraph XC["Cross-cutting"]
    OBS["SLOs · canary · CI/CD"]
    MOD["Model service + secrets"]
    TR["Tool registry (MCP)"]
  end
  GATE --> API
  WK --> DB
  WK --> HOT
  WK --> Q
  WK --> MOD
  WK --> TR
  EV -. gates .-> OBS

The edge platform hosts the UI and an email-gated entry; the cloud region hosts everything that thinks or remembers. Vendor names are deliberately absent from the rest of this chapter: each box is named for the concept it carries, because the concept is what transfers.

Which component does what, and why?

🪤 Misconception — "one store for facts and vectors is a toy choice." At demo scale it is the right choice, and the reason is not convenience. A memory write-back that stores the fact, its embedding and its audit row in one transaction is how the namespace stays a real boundary. The dedicated vector service is a later swap behind the same interface, made when the index, not the design, runs out of room.

How was the control plane made watchable?

The centrepiece is a control plane that implements the desired-state loop of a Kubernetes HPA from first principles, in a form small enough to read. One policy knob — users per instance — sets the ratio; the controller reconciles the running fleet toward the target every time active users or the policy changes.

desired = ceil(active_users / users_per_instance)
while running < desired: start_worker()             # scale OUT
while running > desired: stop_idle_worker()          # scale IN
                        or consolidate_then_stop()   # drain the least-loaded, then reclaim

Two details make it honest rather than decorative. Scale-in is real: when the ratio is raised and every worker is busy, the controller drains the least-loaded worker onto others with spare capacity and only then reclaims it — and rolls the moves back if a drain cannot complete, so an assignment is never lost. And the loop is idempotent: running it twice does nothing the second time, which is the property that lets a real autoscaler run it every few seconds.

stateDiagram-v2
  [*] --> Observe
  Observe --> Compute: active users · policy · triggers
  Compute --> ScaleOut: running < desired
  Compute --> ScaleIn: running > desired
  Compute --> Steady: running == desired
  ScaleOut --> Observe: start worker
  ScaleIn --> Observe: reclaim idle · or drain then reclaim
  Steady --> Observe: tick

The ratio is dialed to one user per instance in the build so the fleet visibly goes 1 → 2 → 3 as personas are switched, and back down as the knob is raised. That is a visualization. The production truth, stated in the fleet chapter, is roughly a hundred users per stateless replica, scaled on aggregate signals — concurrency, CPU, queue depth — which the same policy object also carries as trigger thresholds. The managed autoscaler serving the live deployment is that loop with the ratio replaced by concurrency; a Kubernetes HPA manifest for the same fleet is kept as an artifact so the equivalence is literal, not rhetorical.

⚠️ Pitfall — a control plane that only scales out. Most first attempts add replicas and never remove them, because removing means moving live assignments. A loop that cannot scale in is not a reconcile loop; it is a ratchet, and the bill shows it.

What was measured?

Every turn writes one record — tokens in, tokens out, cost, latency, model, retrieved chunk ids, tool calls, worker id, memory namespace — and an observability panel computes the operating numbers from those records.

The eval result deserves a plain reading. The gate was set at 0.70 on both axes, and the run's mean substance score landed just under it. That is the correct outcome for a regression gate on a new build: it discriminated. Tone (the empathy axis, scored against each persona's tone template) passed comfortably; substance on a few cases did not, and those cases pointed at retrieval choices, not at the model. A gate that everything sails through the first time is usually measuring nothing.

flowchart LR
  GS["Golden set · 9 cases · 3 personas"] --> RUN["Run each case through the live turn"]
  RUN --> J["Judge · substance vs expected · empathy vs tone target"]
  J --> G{"both ≥ 0.70?"}
  G -- yes --> PASS["pass · record run"]
  G -- no --> BLOCK["block deploy · record run"]

turn 3c9a · user maya · worker w-agent-001 · namespace maya
profile: {allergy: peanut, location: SF, tone: warm}
memory: [m12 allergy, m07 short-visual]  general: [g3 weeknight-recipes]
model: small tier · tokens 1.9K in / 0.3K out · latency 2.4 s · fallback: false

What are the honest limits?

• A demo-scale fleet. The teaching control plane manages in-process worker objects in one tier; it proves the reconcile logic, not the operational load of a thousand replicas. The production fleet is the managed autoscaler's, and it was exercised at the traffic of a gated site, not a public launch.
• A single region. One cloud region, one relational store, one hot store. Data residency, active-active regions and sharding by user are designed and documented, not deployed.
• Recorded fixtures on the public site. This knowledge base is static. Where it shows a fleet scaling or an eval run, the numbers are recordings from the build, not a live call.
• A small golden set. Nine cases across three personas is enough to show a discriminating gate, not to characterise a model. The judge is itself a model; its variance is a known limitation.
• Three fictional users. Personalization was proved across three personas in two segments. Real segment discovery — clustering behaviour into tone templates — was not part of the build.
• Scale-to-zero cold starts. Cost discipline means the compute parks at zero between sessions; the first turn after idle pays a cold-start penalty a consumer product would engineer away with a minimum floor.

🎯 At consumer scale, the build's shape holds and its numbers do not. A lab with 100M weekly users keeps the same lanes — edge, stateless compute, sharded state, async, cross-cutting — but replaces every single-instance box with a sharded, replicated, regional one, and turns every hand-tuned knob (ratio, tier, budget) into a policy that a control loop moves. The reference build's claim is narrow and defensible: from one user to a billion is more replicas, more shards and more regions — not a redesign.

🧭 Enterprise mapping. The same build becomes an enterprise reference by attaching the constraints from the previous chapter at seams that already exist: the edge gate becomes SSO against a tenant directory, the memory namespace gains a region and a role list, the eval run is retained as compliance evidence, the tool registry gains per-tenant entitlements, and a human-in-the-loop queue sits between the agentic loop and the stream. None of the lanes move.

Start at the beginning with Start Here, or go straight to One Turn to follow the nine stages this build runs. Every term above has a card in the glossary.