Memory — how an assistant remembers a person, and a billion of them

A person tells their assistant something once — an allergy, a habit, a project they are anxious about — and expects it to still be true next week. That expectation is the whole product. This chapter is how a system keeps that promise for one person, then for a billion, without one person's memory ever touching another's.

The thesis, said this chapter's way: memory is not a longer transcript. It is two stores with two access patterns, a hard boundary per person, a write path that never slows the answer, and a read path that ranks by relevance and recency. Scale is then a storage question, not a redesign.

Why are there two kinds of memory?

Because two kinds of question are being asked of the store, and they want opposite guarantees.

The rule that falls out: identity is never left to similarity. A fact that must be true on every turn is stored where it is returned on every turn; anything merely likely relevant is stored where relevance is computed. Personalization shows how the two fuse into one context pack; this chapter is about the stores behind them.

🪤 Misconception — "one big vector index for everything the user ever said; let the ranker sort it out." Similarity finds the five most related episodes well. It cannot make sure the allergy is present. Rankers have no notion of must.

Where does one person's memory end?

At the memory namespace. Every memory row carries its owner's identifier, and every retrieval is filtered by that identifier before it ranks anything. A worker serving Maya cannot phrase a query that returns Raj's rows: the store applies the filter; the caller is not trusted to.

This is the most important line in the whole memory design, for three reasons:

• Privacy — the namespace is the data boundary; the trust chapter builds on it.
• Performance — a scoped query ranks a few thousand rows, not a few billion. Per-user memory stays cheap even with an exact scan, long after the shared corpus needs specialized indexing.
• Scale — a boundary already keyed by user can be moved. Sharding by user (below) only decides which machine holds a namespace; the query never changes.

Shared knowledge — how-tos, facts about the world, product documentation — lives in a separate corpus with no user filter, because it belongs to nobody. A personal fact can therefore never leak into shared results, and since scores from two corpora are not comparable, the pack budgets each separately rather than merging them into one sorted list.

flowchart LR
    W[Stateless worker] -- "user id → namespace filter" --> M[(Maya's namespace)]
    W -. never .-> R[(Raj's namespace)]
    W -- "no filter" --> C[(Shared corpus)]
    M --> P[Context pack: personal budget]
    C --> G[Context pack: general budget]

How does memory get written without slowing the answer?

Through write-back, off the hot path. The person is already reading a streamed reply while the system decides what, if anything, this turn taught it.

flowchart LR
    T[Turn answered] --> S[Reply streams to the person]
    T -. enqueue .-> Q[(Async queue)]
    Q --> X[Extract durable facts]
    X --> D{Already known?}
    D -- exact or near-duplicate --> K[Skip]
    D -- new --> E[Embed]
    E --> I[(Write into the user's namespace)]
    I --> N[Visible to the next turn]

Each stage has a reason to exist:

• Async queue. Extraction calls a model; embedding calls another. Neither belongs in the 900 ms budget of one turn. Queue depth becomes the write side's scaling signal, independent of the chat fleet.
• Extract, don't transcribe. A small model plus rules pulls durable facts ("prefers weekend hikes"), capped at a handful per turn. Most turns teach nothing, and that is correct.
• Dedup twice. Exact text, then near-duplicate by embedding. "I like hiking" said five times must not become five rows crowding out the one new thing.
• Per-person ordering. Writer and reader are the same loop across turns, so a race does not throw an error — it silently degrades memory. A short per-user lock serializes the write; the next turn's session step waits on it rather than reading stale state.
• Hot mirror. The freshest facts are also mirrored into the session so the very next turn sees them before the index catches up.

⚠️ Pitfall — running write-back inline "just for now." Harmless at ten users; at a million it puts two extra model calls on every response, and p95 latency becomes the write path's problem.

How does the read path choose what to recall?

Two lookups run in parallel, a re-score follows, and a budget decides what survives.

flowchart TB
    Q[Rewritten query] --> K[Keyed lookup: profile facts]
    Q --> V[Vector search inside the namespace]
    V --> D[Re-score with temporal decay]
    D --> B1[Personal budget: top-K]
    Q --> G[Shared corpus search]
    G --> B2[General budget: top-K]
    K --> P[Context pack]
    B1 --> P
    B2 --> P

Temporal decay is the load-bearing detail. Raw similarity has no clock: a preference stated yesterday and a contradictory one from three years ago score identically if they use the same words. So the read path multiplies similarity by an exponential of age, with a gentle λ — recency breaks ties without erasing history:

hits  = vector_search(namespace=user, query_vec, k=6)
for h in hits:
    h.final = h.similarity * exp(-LAMBDA * h.age_days)
personal = top(sorted(hits, by=final), k=4)

The budget is the other half. Because personal and shared scores are not calibrated against each other, the pack reserves slots for each (say three personal, two general) instead of sorting a fused list, so a brilliant shared document cannot evict the one personal fact that made the answer feel written for this person. How the split is trimmed to a token budget is in personalization.

Why does memory compound — and why must it be pruned?

Every useful turn adds a row; every later turn can recall it. Over months the assistant genuinely knows the person, at no extra cost per turn, because retrieval is top-K, not "everything." That is compounding, and it is the moat of a personal assistant.

Unbounded growth is the flip side. Dedup slows it; it does not stop it. Production systems add a periodic memory consolidation pass per namespace: merge overlapping episodes, retire rows whose decayed score can no longer reach the budget, and promote a fact that keeps recurring ("prefers short answers") from episodic memory into the profile, where it is guaranteed rather than ranked. It runs on the same async queue as write-back, at a slower cadence.

Forgetting must be as deliberate as remembering. A "forget me" request deletes the namespace — every episode, every written-back fact, the session mirror — as a single operation, because a memory that survives deletion in one store is a privacy incident, not a bug.

🎯 At consumer scale the write side is its own fleet. A hundred million people generating a few durable facts a day is a steady stream of extraction and embedding calls that never touches the chat path. It scales on queue depth, tolerates minutes of lag, and runs on the cheapest adequate model — because nobody is waiting on it.

How does the store grow from one person to a billion?

In three moves, each with a specific trigger — and the query the worker runs does not change at any of them.

Notice that the shared corpus and the per-user namespaces graduate at different times, for different reasons — which is why keeping them separate on day one matters. And sharding by user is a routing change, not a schema change: a stateless worker already resolves user id → namespace; sharding adds namespace → shard, and the reference architecture shows that same lookup selecting a region.

🧭 Enterprise mapping — a memory namespace becomes a tenant data boundary, and the filter-before-rank rule is what an auditor asks to see. Write-back becomes a governed pipeline: what may be extracted is a policy, retention is a contract, and "forget me" is a legal obligation with a deadline. Sharding by user becomes sharding by tenant with region pinning — how residency promises are kept. The shared corpus becomes the enterprise knowledge base with its own ingestion, ranking and access control, and the rule that it never holds a person's private facts is enforced, not assumed.

Where next

Memory answers what a worker pulls for a person. The next chapter answers how a fleet of identical, stateless workers serves everyone at once, and how the control plane keeps exactly as many running as the moment needs. For the cost of the model calls write-back and recall make, see model serving; for keeping recalled memory from being confidently wrong, see quality and the glossary.