The Fleet — stateless workers and the control plane

Every person chatting with an assistant is served by some worker, somewhere. Whether a product survives its own growth turns not on "how fast is one worker" but on "what must a worker know, and who decides how many exist." This chapter answers both: the stateless worker contract, and the control plane that reconciles the fleet to a desired state.

The thesis, said this chapter's way: scale is not a bigger box. It is many identical workers that hold nothing, a shared set of stores that hold everything, and one small loop — the same loop inside Kubernetes' HPA and inside every managed autoscaler — that keeps the count of workers matched to demand in both directions.

What does "stateless" actually promise?

A stateless worker promises that nothing about a person lives inside it between turns. Concretely, the contract has four clauses:

1. Pull, never hold. At the start of a turn the worker resolves the user id, pulls the session, the profile via keyed lookup, and top-K episodic memory from the user's memory namespace, and assembles the context pack. When the turn ends, it forgets all of it.
2. Write to shared stores only. Anything learned goes out through write-back to the memory and session stores — never into a local variable, disk, or cache another worker cannot see.
3. Any worker, any user. Because of 1 and 2, the next turn for the same person may land on a different worker with no loss. Routing can be by least load, region, or model tier — whatever the control plane finds convenient.
4. Replaceable without ceremony. A worker can start, stop, or crash mid-turn and the only cost is one retried turn. Nothing is drained, migrated, or backed up.

flowchart LR
    U[Person on a phone] --> R[Router]
    R --> W1[Worker]
    R --> W2[Worker]
    W1 --> S[(Session store)]
    W1 --> P[(Profiles + memory)]
    W2 --> S
    W2 --> P
    W1 --> M[Model service]
    W2 --> M

Everything the memory chapter built — the namespace, the keyed profile, the async write-back — exists to make clause 1 cheap and clause 2 safe. The worker is stateless because the stores are well designed.

🪤 Misconception — "stateless means the assistant can't remember me." The opposite: statelessness is what lets memory survive a worker being replaced. State that lives in a worker dies with it.

What is the control plane deciding?

One number per tier: how many workers should exist right now. It comes from two inputs — an observation (how many people are active) and a policy (how many people one worker should serve, the users per instance ratio) — and one line of arithmetic:

desired = ceil(active_users / users_per_instance)

Everything else the control plane does is bookkeeping around that line: which worker each active person is assigned to, which workers are idle, and a record of every scale event so observability can show why the fleet changed. The desired state is not a command ("start three workers") but a target ("there should be three"). That distinction is what makes the loop below safe to run forever.

The reconcile loop

The loop runs on every event that could move either input — a person becoming active or going quiet, an operator editing the policy — and on a timer as a safety net. It is short enough to read whole:

def reconcile(tier):
    desired = ceil(active_users(tier) / users_per_instance[tier])
    while running(tier) < desired:
        start_worker(tier)                                # scale out
    while running(tier) > desired:
        victim = idle_worker(tier) or consolidate(tier)   # reclaim idle, else drain the least-loaded
        if not victim: break                              # nothing can be reclaimed safely
        stop_worker(victim)                               # scale in
    return desired, running(tier)

stateDiagram-v2
    [*] --> Observe
    Observe --> Compute: active users + policy
    Compute --> Compare: desired = ceil(active / users_per_instance)
    Compare --> ScaleOut: running < desired
    Compare --> ScaleIn: running > desired
    Compare --> Settled: running == desired
    ScaleOut --> Compare: start a worker
    ScaleIn --> Compare: reclaim idle, else consolidate
    Settled --> Observe: next event or tick

Three properties make this a control loop rather than a script:

• Idempotent. Running it twice changes nothing the second time, so every event can trigger it without coordination.
• Bidirectional. It scales out and in. A loop that only adds is a leak with a good excuse.
• Convergent. Each pass moves running one step toward desired, stopping when they meet or no safe move remains.

Why is scaling in harder than scaling out?

Scaling out is trivial: a new worker has no users and is immediately useful. Scaling in must respect the people currently being served. The loop tries two moves, in order:

1. Reclaim an idle worker. If any worker has no assigned users, stop it. Nobody notices.
2. Consolidate. If every worker is busy but the fleet is still above desired — which happens when the ratio is raised — pick the least-loaded worker, move each of its users onto another worker that still has room under the ratio, and stop it. If even one user cannot be placed, roll back every move and stop trying. The fleet ends one worker above target rather than one user without a worker.

This worker consolidation is what makes "raise the ratio" a real scale-in, and the rollback rule is what makes it safe. Because workers are stateless, "moving a user" is only bookkeeping: the person's next turn lands elsewhere and pulls the same context pack.

⚠️ Pitfall — an autoscaler that kills the busiest worker because it was started first. Age is not load. Reclaim idle first, consolidate second, and never stop a worker whose users have nowhere to go.

How is this the same as HPA and a managed autoscaler?

Exactly the same shape. The names change; the loop does not.

Reading the first row across is the whole lesson: a ratio of work per worker against an observed amount of work. HPA writes that ratio with utilization; the managed autoscaler with concurrency; this chapter with people, because people are what a personal assistant is provisioned for. Anyone who has tuned one can tune the others.

What triggers scaling in production?

Here is the honest paragraph. "One user per instance" is a visualization. It is chosen so that one person joining makes one worker appear and a policy edit visibly consolidates the fleet — the loop becomes something a reader can watch. Production runs the same loop with the ratio around ~100:1 — stateless replicas each serving on the order of a hundred concurrent people — and the observation is not a headcount but aggregate load: CPU, requests per second (QPS), and queue depth. The formula does not change; only the numerator and the policy do. Doing autoscaling on load rather than headcount matters because a hundred quiet people and ten intense ones cost the same machine very different amounts.

Three signals earn a place in the policy:

• CPU / memory — the classic HPA target; catches expensive turns (long context, many tool calls).
• QPS per replica — catches bursts before CPU does, because a queued request is not yet burning CPU.
• Queue depth — the truthful signal for the async fleets (write-back, eval): a growing backlog means add workers, even though no person is waiting.

🎯 At consumer scale the fleet is not one fleet. Chat workers, retrieval workers, write-back workers and eval runners are separate tiers with separate ratios and triggers, each reconciled by the same loop. The control plane's real job is a policy per tier, so a burst of memory writes scales the write tier without touching the chat tier's bill.

From one person to a billion

Because workers are stateless and every per-user query is already scoped to a namespace, growth is "more replicas, more shards, more regions" — not a redesign. The ladder:

Where it actually bottlenecks is never the stateless workers. It is (1) write throughput on the relational primary — sharding by user; (2) model cost and latency — model serving; (3) unscoped vector search over billions of documents — dedicated vector infrastructure; and (4) per-user data residency — region-pinned shards. None of the four touches the loop on this page.

🧭 Enterprise mapping — the reconcile loop is unchanged; the policy gains constraints. Users per instance becomes capacity per tenant with an SLO attached: a contract may guarantee minimum replicas so a tenant's p95 never depends on a neighbour's burst. Region pinning stops being an optimization and becomes a residency clause. Scale-in gains a change window ("never consolidate during the tenant's trading hours"). And every scale event — already a ledger here — becomes audit evidence that capacity was managed as promised. Same chassis, stricter policy.

Where next

The fleet keeps the right number of workers alive. What each worker spends per turn — and why the bill is tokens, not machines — is model serving. How a worker may act on the person's behalf is tools; how the loop is watched, gated and rolled out is quality; the reference architecture shows it running against a real managed autoscaler. If the ladder felt like a lot at once, start here.