A retrieved-exemplar store keeps teaching a product name retired six months ago — how do you fix it?

## Why staleness bites harder here than in a static prompt When demonstrations are hardcoded in a template, a retired product name is in a file someone reads during review, appears in every request, and shows up the moment anyone looks at the prompt. When demonstrations are retrieved, the same stale example is dormant for most traffic and activates only for the queries near it. The result is a regression that is concentrated, intermittent, and absent from an aggregate accuracy number — which is why teams find it through a customer complaint rather than a dashboard. The deeper point an interviewer is testing: adding retrieval to few-shot prompting converts prompt content into *data*, and data needs a lifecycle. Nobody would ship a training set with no timestamps, no provenance and no review path; an exemplar pool is that training set, applied at inference time. ## Metadata is the whole fix Every entry should carry, at minimum: when it was created, who or what verified its label, and the window over which it is considered valid. That last field is what lets you retire the obsolete product without deleting history — you set a validity end date, and the retrieval filter stops returning it. Deletion loses the audit trail and makes it impossible to answer "why did this system say that in March?" With the metadata in place, retrieval becomes a filtered search rather than a raw nearest-neighbour lookup: drop entries outside their validity window, then rank. Recency weighting is a softer version of the same idea — at comparable similarity, prefer the newer case — which handles gradual convention drift where nothing is strictly wrong but older phrasing is worse. ## Curating what goes in Write-back is what makes a pool valuable and what makes it dangerous. The valuable version: when a case is resolved by a human and the outcome is verified, the input and the verified outcome enter the pool. The dangerous version: the system's own output is written back automatically. The second creates a closed loop — the model's mistakes become demonstrations, the demonstrations make the mistake more likely, and the pool converges on its own errors while looking busy and well-maintained. If you must auto-write, mark provenance as unverified, keep those entries out of retrieval until reviewed, and never let an unverified entry become the nearest neighbour for a class of queries. A second admission rule is worth stating: an entry earns its place by being *teachable*, not merely by being correct. Correct-but-idiosyncratic cases teach idiosyncrasy. ## Auditing on a schedule Taxonomies move: queues get merged, categories renamed, products retired, policies rewritten. None of those events touch the pool unless someone makes them. Two mechanisms keep the pool honest. First, a periodic reconciliation against the current label set and product vocabulary — anything referencing a label or entity that no longer exists is flagged for retirement or relabelling. This can be largely automated because it is a set difference, not a judgement call. Second, coverage and staleness telemetry: the age distribution of the entries actually being retrieved, and the share of retrievals hitting entries older than some threshold. A rising median age of retrieved exemplars means the pool is being outrun by the traffic. It is the same shape of signal as a falling similarity score, and it deserves the same alert. ## Versioning and attribution The pool must be versioned, and every request should log the pool version alongside the retrieved ids. Without that, every quality question becomes unanswerable: a change in output could be a model update, a prompt change, or someone adding forty entries to the pool on Tuesday. With it, you can diff versions, roll back a bad batch, and pin an evaluation to a snapshot so your regression suite is measuring the model rather than the drift of its demonstrations. Pinning also protects against the quieter contamination problem: if production inputs are written back and your evaluation inputs came from production, the pool can eventually contain the eval items themselves, turning the evaluation into a lookup. ## The answer in one line Retire the obsolete entries by validity window rather than deletion, filter and recency-weight at retrieval, gate write-back behind human verification, reconcile against the live taxonomy on a schedule, and version the pool so you can attribute and roll back. The product name is a symptom; the missing lifecycle is the defect.