In RAG evaluation, why can context precision look high while answers still miss facts?

## Why the retriever gets two metrics Stage-wise RAG evaluation splits the retriever's job in two, because there are two distinct ways for retrieved context to be wrong: it can be **missing something** or it can be **full of noise**. One number cannot express both, and optimising either alone produces a characteristic pathology. ## Context recall Context recall asks: *does the retrieved context contain everything needed to produce the correct answer?* The usual implementation is reference-based rather than label-based. You supply a ground-truth answer for the query, decompose it into atomic claims, and for each claim ask whether it can be attributed to any of the retrieved chunks. Context recall is the fraction of reference claims that can. A model typically performs the attribution judgment, which is what makes the metric cheap enough to run at scale. The practical consequence is that you need *reference answers*, not per-chunk relevance labels — a materially different annotation cost from classical recall@k, and often an easier one, because writing the correct answer to a question is more natural for a domain expert than judging fifty documents. Low context recall means the pipeline is information-starved. No prompt engineering, no stronger generator and no reranking can fix it, because the facts are not in the window. ## Context precision Context precision asks: *of what was retrieved, how much was useful, and was the useful part near the top?* It is computed per retrieved chunk — is this chunk relevant to answering the question, judged either against the reference answer or by a model — and then combined as a rank-weighted average, which is an average-precision-style aggregation. Two things therefore move it: the proportion of useful chunks, and their positions. Ten chunks of which two are useful score badly; the same ten score better when those two are ranked first. Low context precision means you are paying tokens for noise and giving the generator plausible material to be misled by. It rarely makes an answer impossible, which is exactly why it is the less alarming of the two. ## The asymmetry, concretely A legal contract-review assistant is asked: "does the termination-for-convenience clause survive assignment of the agreement?" Retrieval returns three chunks: the termination-for-convenience clause, a definitions chunk covering "Assignment", and a governing-law clause. Two of three are on point and ranked first and second, so context precision is high. But the carve-out stating that termination rights do not transfer on assignment lives in a schedule at the back of the contract that was never indexed as related. The reference answer contains a claim that no retrieved chunk supports, so context recall drops — and the generated answer will be confidently incomplete. The reverse failure is just as real: retrieve thirty chunks and you will probably cover every claim (high recall) while burying them among irrelevant text (low precision), producing a bloated, expensive prompt and a higher chance of the model anchoring on the wrong passage. ## Reading them together - **Low recall, any precision** — the ceiling problem. Look at chunking, coverage of the index, embedding fit, query rewriting, or the k you retrieve at. Fix this first; nothing downstream matters until it is fixed. - **High recall, low precision** — a ranking and cutoff problem. Rerank, tighten the shipped k, or filter by score. Cheap to fix and mostly a cost issue. - **Both high, answers still wrong** — retrieval has done its job and the defect is on the generation side. That is the whole point of splitting the metrics: it tells you to stop debugging the retriever. ## Caveats worth stating in an interview These metrics are model-computed, which buys scale at the cost of some noise: attribution judgments vary run to run, and a claim that is *implied* rather than stated in a chunk sits in a genuine grey zone. Pin the judging setup and treat small movements as noise rather than progress. They also depend entirely on the quality of the reference answers — an incomplete reference makes context recall look better than the system deserves, because the claims it omits are never checked. Finally, note the relationship to classical IR metrics. Context recall and recall@k measure the same intuition from different directions: recall@k needs labelled relevant chunks, context recall needs a reference answer. Teams often run classical metrics during retriever development, where chunk-level labels exist, and the reference-based pair in continuous evaluation, where writing answers is the cheaper annotation.