Rerank Cascades and Result Selection

## The cascade as three separate numbers A retrieval-augmented generation (RAG) pipeline that reranks has at least three tunable widths: **k**, how many candidates the cheap first-stage retriever returns; **n**, how many of those the expensive reranker actually scores; and **m**, how many survivors are written into the prompt the model reads. They are frequently conflated ("we use top-5"), but each is governed by a different constraint, measured with a different signal, and fails in a different way. ## Choosing k — recall is the ceiling First-stage retrieval exists to be cheap and generous. Its job is not to rank well; its job is to not lose the answer. Whatever recall you have at k is a hard ceiling on the whole pipeline: a reranker can only reorder what it was given, so a passage missing from the candidate pool is unrecoverable no matter how good the later stages are. The practical method is a sweep. Take a labelled set of queries with known-relevant passages, and plot the fraction of queries whose relevant passage appears anywhere in the top k, for k = 10, 25, 50, 100, 200. The curve rises steeply then flattens. Pick k somewhere past the knee — the point where doubling k buys a percentage point or less. Typical production values sit between 50 and 200 for a corpus of any size; very small corpora may need far less. ## Choosing n — the reranker's cost curve The second stage is expensive per candidate, so its cost is roughly linear in n. That makes n the budget dial. Two constraints bound it from above: the latency you can afford before the user sees a first token, and the money per query. Two observations bound it from below: n smaller than the k where recall flattened throws away recall you already paid for, and an n so small that the reranker is only reordering near-ties gives you no measurable quality lift. When a single n cannot satisfy both quality and latency, you add a stage rather than compromise: 200 candidates → a cheap intermediate scorer → 25 → an expensive reranker → 5. Each stage should be strictly cheaper per item and strictly higher-recall than the one after it. More than three stages is rare outside web-scale search; each stage adds a place for tuning to rot. ## Choosing m — what the reader can actually use m is not "as many as fit". Long contexts degrade: attention over a large body of loosely relevant text makes the model likelier to lean on a weak passage, to blend two documents into a claim neither made, or to miss the one passage that mattered. Every extra passage also costs tokens on every request forever. In practice m is small — often three to eight chunks — and is chosen by running the end-to-end answer eval at several values and taking the point where answer quality stops improving, not by filling the window. m interacts with chunk size. Twenty small chunks and five large ones can be the same token count with very different behaviour, so tune m against the chunking you actually ship. ## A tuning procedure that works 1. Fix chunking and the retriever. Sweep k, measure candidate-pool recall, choose k past the knee. 2. Fix k. Sweep n against latency and cost per query; confirm end-to-end answer quality is flat above your chosen n. 3. Fix k and n. Sweep m against end-to-end answer quality, not ranking quality, since m is the only number the generator sees. 4. Re-run the sweep whenever chunking, the embedding model, the reranker, or the corpus composition changes. These numbers are not portable across those changes. ## Failure modes to name in an interview - **k too small.** Ceiling failure. Symptom: the reranker looks fine on the candidates it gets, but the system confidently answers from the wrong document. Diagnostic: candidate-pool recall, measured separately from final answer quality. - **n ≈ k with a large k.** You are paying full reranker price for the entire pool; latency climbs linearly while quality is flat past the knee. - **m too large.** Token cost rises, answers get vaguer, and the true passage competes with distractors. This is the most common silent regression, because ranking metrics improve while answers get worse. - **Tuned on the wrong signal.** Optimising k and n on ranking quality alone is fine; optimising m that way is not — only an end-to-end answer eval can tell you whether the reader benefited. - **Never re-tuned.** Numbers chosen for a 10k-document pilot are usually wrong at 10M documents. The interview answer that lands is the one that says these are three decisions with three different measurements, gives a concrete shape such as 100 → 25 → 5, and names recall as the ceiling everything else lives under.