How would you measure whether an agent's memory retrieval is actually helping?

## Recall is the wrong headline metric Classic retrieval metrics — recall@k, precision@k, MRR against a labelled set — measure whether the store can surface a memory someone already declared relevant. Three things make that a weak proxy for an agent. The labels are usually written by the same people who designed the store, so they encode the retrieval strategy rather than testing it. The metric is blind to whether the model *used* what it received; a perfectly retrieved memory that the model ignores has bought nothing. And it counts only misses, never harm — retrieving a stale fact is scored as a success if the fact was labelled relevant, even when it makes the agent confidently wrong. ## The funnel worth instrumenting Think of memory as a pipeline with drop-off at each stage, and instrument each stage separately. **Retrieved per turn.** Trivial to log, and the denominator for everything else. Also log tokens spent on memory, because that is what you are trading. **Usage rate.** Of the memories that entered context, how many actually influenced the reply? The cheap mechanism is attribution: give each memory a short identifier and require the agent to cite the ones it relied on. It is imperfect — models under-cite and occasionally cite decoratively — but it is directionally reliable and nearly free. The costlier mechanism is a judge over the trace, asking whether the reply depends on a given memory. When this number comes back at a few percent, you have learned that most of your memory budget is noise, and that is the single most actionable memory metric there is. **Decisiveness.** Usage does not prove benefit; the model might have answered identically without it. Only a counterfactual settles it. Run the same task set three ways: memory disabled, memory enabled, and memory enabled with a deliberately wrong or stale fact injected. The gap between off and on is what memory is worth. The drop under the poisoned condition is your sensitivity to bad memory, and it is often alarming — a system that ignores memory scores well on the ablation for the wrong reason, and a system that follows any memory blindly is one stale fact away from a bad answer. **Harm.** Track memory-induced errors as a first-class rate: turns where the agent was wrong *because* of what it recalled. This is the metric that justifies validity filters and scope filters to a business, and it is asymmetric — one confidently wrong answer sourced from a stale fact costs far more trust than one missed recollection. ## Online signals Offline suites go stale against a moving user population, so pair them with production instrumentation. User corrections are the highest-signal event available: when someone says "no, I changed that months ago", tag the turn and trace which memory produced it. Repeat-telling is the mirror image — a user restating something already stored means retrieval missed, and it is measurable by matching new writes against existing memories. Both are cheap, unambiguous and continuously available, which makes them better guardrail metrics than any judge. Beyond that, run memory changes as A/B experiments on the outcome the product actually cares about — task completion, turns to resolution, escalation rate — rather than on retrieval quality. The whole point of the funnel is that retrieval quality is an intermediate. ## Public benchmarks and their limits Long-conversation memory benchmarks exist and are useful for orienting on architecture choices — filesystem-style memory, temporal knowledge graphs and vector stores post materially different scores on them, particularly on questions requiring reasoning about facts that changed over time. Treat them as a coarse filter. They test synthetic dialogues over a fixed distribution, and by mid-2026 the differentiator between systems is private evaluation on the real task distribution, not leaderboard position. ## What to do with the numbers The measurements should drive concrete policy changes, and saying which is what separates a strong answer from a recital. A low usage rate with a high retrieval count argues for cutting k, tightening filters, or moving to agentic recall so the model pulls only what it decides it needs. A high usage rate with no ablation gap suggests the memories are true but redundant — the model would have inferred them anyway — so the store is paying to restate the obvious. A large drop under poisoned memory says the agent over-trusts memory and needs provenance and validity annotations in the context so it can discount. And frequent user corrections point squarely at write and validity policy rather than at ranking. Finally, price it. Memory has a per-turn token and latency cost, and expressing the benefit per thousand tokens spent is what lets you decide between a bigger memory budget and spending those tokens on tools, examples or a stronger model. That framing — memory as one claimant on a finite context budget competing with every other claimant — is the judgment the question is really probing.