In search relevance evaluation, what do precision@k and recall@k measure, and how does k change each?
answer
- Two fractions with different denominators
- One divides by k, the other by everything relevant
- Deepening the list can never lose you relevant hits
- The other one dilutes as you add slots
- Recall's denominator is the hard part to know
basics
~20 sPrecision@k is the fraction of the top k results that are relevant. Recall@k is the fraction of all relevant documents that appear in the top k. Raising k can only raise recall, and usually lowers precision.
solid answer
~50 sBoth score one ranked list against relevance judgments for a query. **Precision@k** = relevant documents in the top k divided by k: how much of what I showed was useful. **Recall@k** = relevant documents in the top k divided by the total number of relevant documents for that query: how much of what exists I actually found. As k grows the numerator can only stay flat or grow, so recall@k is non-decreasing, while precision@k usually decays because the extra slots are mostly non-relevant. That is the trade-off: a shallow cutoff flatters precision, a deep cutoff flatters recall. Neither metric looks at the order inside the top k, so a perfect list and a shuffled one score identically — which is why teams add rank-aware metrics. Recall is also the expensive one, since it needs the full relevant set, which on a large corpus is only ever estimated.
code
python · 6 linestop_k = ["d7", "d2", "d9", "d1", "d4"] # ranked results
relevant = {"d2", "d4", "d11", "d15"} # judged relevant for this query
hits = len([d for d in top_k if d in relevant]) # 2
precision_at_5 = hits / len(top_k) # 0.40
recall_at_5 = hits / len(relevant) # 0.50go deeper
Be ready to state both formulas from memory and compute them on a small worked example, including remembering that the two denominators differ.
Explain why recall@k is monotonic in k while precision@k is not, and pick a sensible cutoff for a given interface rather than defaulting to 10.
Show judgment about which metric a workload should be optimised for, and be explicit that recall's denominator comes from a pooled estimate that biases the number.
Own the framing that different pipeline stages need different metrics and different cutoffs, and that publishing one headline number invites teams to optimise the wrong end of the trade-off.
## The setup Every offline search metric needs two inputs: a ranked list the engine returned for a query, and a set of **relevance judgments** saying which documents are relevant to that query. Precision and recall are the oldest way of turning those into a number, and nearly every richer metric is built on top of them. ## Precision@k Precision@k counts relevant documents among the first k results and divides by k: `precision@k = |relevant ∩ top-k| / k` If the first ten results contain three relevant documents, precision@10 is 0.3. It is cheap to compute because you only need judgments for the documents you actually showed. It maps directly onto user experience on a result page: a user scanning ten blue links experiences precision@10 fairly literally. Its blind spot is that it says nothing about what you missed. A search engine that returns exactly one result, and that result is relevant, has precision@1 of 1.0 while ignoring hundreds of other relevant documents. ## Recall@k Recall@k divides by the size of the whole relevant set R for that query: `recall@k = |relevant ∩ top-k| / |R|` If the corpus contains twelve relevant documents and three of them are in the top ten, recall@10 is 0.25 — note that the same list gives precision@10 = 0.3 and recall@10 = 0.25. The denominators differ, so the two numbers are not comparable to each other, only across systems. Recall matters most when missing a document is expensive: legal e-discovery, patent search, compliance, recruiting, or any first-stage retrieval that feeds a re-ranker. In those cases a document not retrieved at stage one can never be recovered later, so first-stage recall is a ceiling on the whole pipeline's quality. ## Why k changes each differently Walk down the ranking one position at a time. Each new position either adds a relevant document or does not. - Recall@k is **monotonically non-decreasing** in k. The numerator can only grow and the denominator is fixed, so deepening the cutoff never hurts recall. At k = corpus size, recall is 1.0 by construction, which is why an unqualified "recall" number is meaningless without a cutoff. - Precision@k has no such guarantee, but in practice it **decays**, because a decent ranker puts its best guesses first. It rises only when a relevant document appears at a position deeper than the current average density. That asymmetry is the classic trade-off. Any change that pulls more candidates in — loosening a filter, adding synonyms, lowering a minimum-match threshold — tends to move recall up and precision down. Tightening does the opposite. ## Choosing k k should be an honest model of what users see or what the next stage consumes. - A ten-result page: report precision@10, often also precision@1 and precision@3, because eye-tracking and click data both say attention concentrates at the top. - An autocomplete or answer box that shows three suggestions: k = 3. - A candidate generator feeding a re-ranker that re-scores 200 documents: recall@200 is the number that matters, and precision there is almost irrelevant. Reporting several cutoffs is normal and cheap; reporting only one hides the shape of the change. ## The measurement problem Precision@k needs judgments only for documents you showed. Recall@k needs |R|, the count of every relevant document in the collection, which nobody can enumerate for a web-scale or even enterprise-scale corpus. In practice |R| is approximated from a pooled judgment set: run several systems, judge the union of their top results, and treat that union as if it were the complete relevant set. That approximation is where most of the error in a reported recall number lives, and it systematically under-counts, so absolute recall figures should be read as a comparison between systems evaluated on the same judgments, never as a physical truth. ## Combining them Because the two move in opposite directions, teams often summarise with **F1**, the harmonic mean of precision and recall, which punishes a system that wins one by sacrificing the other. F1 is common in classification and set-retrieval settings but less useful for ranked search, because it still ignores order. That is the natural bridge to rank-aware metrics: MRR, MAP, and NDCG all reward putting relevant documents *higher*, not merely inside the cutoff. ## What an interviewer is listening for The definitions, the direction each moves as k grows, one concrete workload where recall dominates and one where precision does, and the awareness that recall's denominator is an estimate rather than a fact.
- Why is precision@k a poor metric for comparing two rankers that return the same ten documents in different orders?Precision@k is set-based inside the cutoff — it counts relevant documents in the top k and ignores their positions. Two lists containing the same three relevant documents score identically whether those documents sit at ranks 1-3 or 8-10, even though users experience them very differently. Rank-aware metrics such as MRR, MAP, or NDCG apply a positional discount and separate the two.
- For a two-stage system where a cheap retriever feeds an expensive re-ranker, which metric do you put on each stage?Measure the retriever with recall at the candidate depth the re-ranker consumes — recall@100 or recall@1000. Anything the first stage misses is unrecoverable, so its recall is a hard ceiling on the whole pipeline. Measure the final ranking with a rank-aware precision-flavoured metric such as NDCG@10, since that is what the user actually sees.
- Why can recall@k be reported as 1.0 and still be misleading?Because the denominator is usually a pooled estimate of the relevant set, not the true set. If the judgment pool was built only from systems similar to the one under test, every relevant document it knows about is one your system already finds, so recall looks perfect while genuinely relevant unjudged documents sit unretrieved and uncounted.
Fishing with a net: precision is the share of your catch that is edible fish, recall is the share of the lake's fish you caught. A bigger net always catches more fish and more rubbish.
saying these in an interview costs you the question
- Saying recall can drop when you increase k
- Treating precision@k and recall@k as comparable numbers
- Claiming recall is easy to compute on a large corpus
- Using precision@k to compare two orderings of the same results
- Reporting a single unqualified 'recall' with no cutoff