In RAG retrieval evaluation, how do recall@k and precision@k differ, and which one caps answer quality?

## The two definitions, side by side Both metrics look at the same object: the ordered list of chunks a retriever returned for one query, truncated at position k. What differs is the denominator. - **recall@k** = (relevant chunks in the top k) / (all relevant chunks that exist for this query). It answers *how much of what I needed did I get?* - **precision@k** = (relevant chunks in the top k) / k. It answers *how much of what I got did I need?* Both are per-query numbers, averaged over an evaluation set. A single query with six relevant chunks, of which three appear in a top-10 list, scores recall@10 = 0.5 and precision@10 = 0.3. The two move against each other as k grows. Increasing k can only add relevant chunks, so recall@k is monotonically non-decreasing in k; precision@k usually falls, because the tail of the ranking is mostly noise. This is why quoting either number without its k is meaningless. ## Why recall is the ceiling in RAG RAG has a strict information bottleneck: the generator sees only the chunks retrieval placed in the prompt. Everything downstream — reranking, prompt design, a stronger model, self-critique — can only reorganise, ignore or summarise that set. It cannot conjure a fact that was never retrieved. Concretely, imagine an insurance claims-handling knowledge base. An adjuster asks whether a windscreen replacement is subject to the standard deductible. The rule lives in one clause of the policy-exceptions document. If that clause is not in the top k, the best possible outcomes are a refusal or a confident answer derived from the general deductible chunk — which is wrong. Retrieval recall has capped the system at *wrong or silent*, and no generation-side metric will explain why. That is precisely why RAG evaluation is split by stage: a low retrieval recall localises the bug before anyone starts blaming the model. ## Why precision still matters It is tempting to conclude that you should just make k huge and stop worrying. Three costs push back. 1. **Token and latency budget.** Context is paid for on every request, and long prompts raise time-to-first-token. 2. **Distraction.** Models are demonstrably pulled off course by passages that are topically close but factually wrong — a superseded policy version, a chunk about a different product line. A plausible distractor is more dangerous than an obviously irrelevant one. 3. **Attention dilution.** As the retrieved set grows, the signal-to-noise ratio of the prompt falls, and the relevant sentence competes with more surrounding text. So the practical target is: get recall high at a generous k, then raise precision *at the k you actually ship* by reordering, not by retrieving less. ## Multi-chunk questions and the hit-rate trap **Hit rate** (sometimes called success@k) scores 1 if *any* relevant chunk is in the top k, and 0 otherwise. It is the crudest of the family and it is fine when every question is answerable from exactly one chunk. Most real corpora are not like that. A question such as "is a windscreen claim excluded, and what is the notification window?" needs two clauses. Hit rate awards a perfect 1.0 for retrieving one of them, hiding a systematic failure. Per-query recall@k, with the full required set as the denominator, exposes it: that query scores 0.5 and the average drops. Whenever a corpus supports multi-hop or composite questions, prefer recall@k over hit rate and make sure the ground-truth set lists *every* chunk a correct answer needs. ## Reading them together A useful diagnostic pattern: - **Recall@k low at every k** — the failure is upstream of ranking: chunking that splits the answer, an embedding model that does not understand the domain vocabulary, or queries whose phrasing never matches the documents. Fix retrieval inputs, not the ranker. - **Recall@k high at a large k but low at a small k** — the right chunk is in the candidate pool but ranked badly. This is a ranking problem, and a reranker is the cheap fix. - **Precision@k low while recall@k is fine** — you are paying tokens for noise; shrink the shipped k or rerank. One last caution: both metrics assume your relevance labels are complete. If a genuinely relevant chunk is unlabelled, it is counted as a false positive and precision is understated. Incomplete judgments make precision the less trustworthy of the two numbers on a hand-built set.