Memory Retrieval Policies

## Two ways a memory reaches the prompt **Pre-injection (retrieve-then-prompt).** Before the model is invoked, the harness takes the user's latest message, searches the memory store, keeps the top k results — usually five to ten — and places them in the system prompt or a dedicated memory block. The model never asks for them; they are simply there. This is the classic pipeline shape borrowed from retrieval-augmented question answering. **Agentic recall.** The memory store is exposed to the model as a tool: a search call, a read call, or a plain directory the agent can list, open and grep. Mid-turn, the model notices it lacks a fact, issues a call, reads the result and continues. Anthropic's file-directory memory tool and Letta's filesystem-style memory are both this shape, and by mid-2026 it is the dominant pattern for long-horizon agents, with blind top-k pre-injection retreating to the small always-on profile. ## What pre-injection buys, and what it costs It buys a single round trip, which matters enormously when the product has a hard latency budget — a voice assistant cannot afford a recall turn before it starts speaking. It is deterministic and therefore easy to evaluate and replay: the same conversation state yields the same injected memories. It costs on three fronts. First, it is unconditional: it pays retrieval cost and context tokens on every turn, including the large majority of turns that need no memory at all. Second, the query is whatever the user just typed, which is often a terrible retrieval query — "and the other one?" carries no searchable content, so the top k is effectively random. Third, fixed k has no notion of "nothing relevant here": if the store holds no useful memory, top-k still returns k items, and those near-misses sit in the window as plausible-looking noise. That is the mechanism behind context rot and lost-in-the-middle degradation, and it is why more memory can make an agent worse. ## What agentic recall buys, and what it costs Cost becomes conditional — nothing is spent on turns that need nothing. The query is model-authored, so it can be a resolved, explicit query ("the appointment-time preference recorded for this patient") rather than the user's fragment. The model can iterate: search, find nothing, widen the query, or list a directory and open only the two files that look right. It can also filter and aggregate before anything enters context, which is exactly what makes a grep over a `memory/patients/` directory cheaper than a pre-injected top-8: the agent reads one file instead of eight snippets. And the model can conclude "I have no memory of this" and say so, which fixed-k retrieval structurally cannot express. The costs are real. Every recall is an extra model turn plus a tool round trip, so a turn that needs two lookups can triple its latency. The model may fail to look at all — an agent cannot retrieve a fact it does not suspect exists, and this silent miss is the characteristic failure of agentic recall. It is also nondeterministic, which makes regression testing harder: two runs of the same conversation may retrieve different things. ## Choosing, in practice A workable default is a split by size and hit rate. Pre-inject the small, high-hit-rate core — identity, language, standing preferences, active goals — because a few hundred tokens that are relevant most turns are effectively free and save a round trip. Make everything else agentic: the long tail of episodic detail, per-entity files, older sessions. Then tune with two numbers. If a large fraction of turns actually consume an injected memory, injection is earning its tokens. If the fraction is small — and it usually is — agentic recall is cheaper and cleaner. On the other side, if the agent frequently answers as though a stored fact does not exist, the problem is discovery, not the store: fix it by pre-injecting an index or table of contents of what memory holds, so the model knows what is worth asking for. Latency decides the edge cases. Real-time voice and autocomplete cannot absorb a recall turn; asynchronous agents doing multi-minute work absorb several without anyone noticing. ## Failure modes worth naming Blind top-k on a memory-irrelevant turn, poisoning an otherwise clean context. A model that never calls the memory tool because nothing in the prompt suggests memory exists. Recall loops, where the agent searches repeatedly with near-identical queries and burns its budget. And unbounded results, where an agentic read pulls a whole file that a pre-injected snippet would have trimmed — agentic recall removes the k cap, so the tool itself must impose size limits.