What is dynamic exemplar retrieval in few-shot prompting, and how does it differ from a fixed example block?

## The idea Few-shot prompting teaches a task by showing worked examples rather than describing the rule. The usual form is *static*: an engineer picks a handful of input/output pairs, pastes them into the prompt template, and every request sees the same demonstrations. **Dynamic exemplar retrieval** replaces that fixed block with a per-request selection: you keep a pool of labelled examples, and for each incoming input you retrieve the nearest ones and use those as the demonstrations. It is sometimes called kNN few-shot prompting, because the selection is a k-nearest-neighbour lookup over a similarity space. ## The pipeline Offline, you build the pool: input/output pairs whose outputs are *verified correct*, each turned into a vector by an embedding model and stored in a searchable index. (Which embedding model and which index structure you use is a separate subject with its own trade-offs.) Online, each request does four things: embed the incoming input; search the pool for the k most similar entries; format those entries as demonstrations in the prompt's expected input/output shape; call the model with the assembled prompt. Everything except step three is standard retrieval machinery — what makes it exemplar retrieval rather than document retrieval is *what the retrieved items are for*. In retrieval-augmented generation, you retrieve documents because they contain the facts the answer needs. Here, you retrieve labelled pairs because they demonstrate the *behaviour* the answer should imitate; the retrieved content is not evidence, it is a pattern. ## Why per-request selection helps A fixed block has to be representative of the entire input distribution at once. That is easy when the task is narrow — binary sentiment, one output shape — and hard when it is not. The wins concentrate in three situations: - **Large label spaces.** Routing a support ticket into one of 200 queues, or tagging against a deep product taxonomy, cannot be demonstrated in five fixed examples. Retrieval delivers demonstrations from the neighbourhood the query actually lives in. - **Idiosyncratic conventions.** Where the correct answer depends on local house rules the model cannot infer — how *your* team words a severity field, how *your* analysts phrase a rationale — the nearest prior cases carry those conventions implicitly. - **Long tails.** Rare input shapes are, by definition, not in a small fixed block, but they are usually in a large pool. Retrieved demonstrations also tend to share vocabulary with the query, which nudges the model toward the right register and the right label wording without extra instruction. ## What it costs **Latency.** Embedding the query plus the index lookup happens before the model call, adding to time-to-first-token on every request. **Prompt caching.** Caching depends on an unchanging token prefix. A block that differs per request breaks that prefix, so the tokens from the block onward are re-processed every call. Where you place the block therefore has direct cost consequences. **Ownership.** The pool becomes a production dependency: it needs labels you trust, a write path for new cases, and deprecation of cases that have gone stale. A wrong label sitting in the pool silently teaches the wrong rule to every query that lands near it. **Reproducibility.** The same input can produce different output tomorrow because the neighbourhood changed. Offline evaluation has to pin a pool snapshot, and incident debugging has to log which exemplars were retrieved, or you cannot tell whether the model or the neighbourhood moved. **Neighbourhood risk.** Nearest is not the same as representative. A neighbourhood can be all one label, or a set of near-duplicates carrying one example's worth of information, or — for a genuinely novel input — nothing close at all. ## When a static block still wins When the input space is homogeneous and the label space small, retrieval buys little and costs a hop. When the prompt is long, stable and cache-friendly, the cache economics can dominate the accuracy gain. When latency budgets are tight, the extra round trip may not be affordable. And when the pool is small, the nearest k are approximately the same examples every time — you have paid for infrastructure to reproduce a static block. ## How you decide Treat it as an experiment, not an architecture decision. Build the pool, run the retrieved variant against the fixed block on a held-out set, and look at the lift *per segment* rather than in aggregate: dynamic retrieval usually shows a modest average gain concentrated in the long tail. Then weigh that lift against the added latency, the cache loss, and the ongoing curation burden before shipping it.