In hybrid retrieval, how deep should each leg fetch before the lists are fused?

## Two different k's Hybrid pipelines carry two numbers that people routinely conflate: - **Per-source top-k** — how many candidates each retriever returns before fusion. - **Final k** — how many fused results the caller keeps. They are not the same parameter and should not be given the same value. Per-source k is a *recall* budget; final k is a *precision and payload* budget. ## Fusion cannot invent candidates The hard constraint: a merge step can only reorder documents it was given. If a relevant document sits at rank 63 in the dense list and rank 9 in the lexical list, and you fetched 10 from each, only the lexical appearance exists and the fused evidence is one term. Fetch 100 from each and the same document accumulates two contributions and rises. Whatever neither leg returned is unrecoverable at any later stage. That makes per-source k the ceiling on the whole retrieval pipeline's recall — the most consequential number in the retrieval stage, and the cheapest one to get wrong invisibly. ## The truncation trap Setting per-source k equal to final k has a specific, damaging effect under rank fusion. Fused scores are dominated by cross-list agreement: a document in both lists collects two terms, one in a single list collects one. Under RRF with k=60, a document at rank 20 in both lists (2/80 = 0.025) outranks a document at rank 1 in only one list (1/61 ≈ 0.0164). That is usually the behaviour you want — consensus is evidence — but it means single-leg finds are systematically pushed down. When the fetch window is shallow, those single-leg finds fall off the end and you lose exactly the documents hybrid retrieval was adopted to catch: the identifier match only BM25 found, the paraphrase only the embedding found. Fetch deeply and truncate late, so demotion is a ranking decision rather than a window artifact. ## Sizing it empirically The measurement is straightforward. On a labelled query set, compute per-leg recall@k as k grows: 10, 20, 50, 100, 200. Each leg's curve rises and then flattens; the useful depth is where flattening begins, per leg, per query slice. Then check fused quality at the final cut, since more candidates is not monotonically better downstream — a longer candidate list means whatever consumes the fused output sees more marginal documents. Typical outcomes: the leg that is strong on your traffic saturates early (its relevant documents are near the top anyway), and the weaker leg keeps contributing further down, which is an argument for asymmetric budgets rather than one shared number. Asymmetry is also how you express a preference without touching a fusion weight — depth is a blunt but very stable knob. ## What the merged tail looks like A useful diagnostic when you extend depth: inspect the composition of the fused list by position. Near the top, documents tend to appear in both source lists — agreement is concentrated there. Deep in the tail, nearly every entry appeared in exactly one list and carries a single small reciprocal term, so those positions are close to arbitrary ordering between near-tied scores. Once your tail is entirely singletons at essentially tied fused scores, extra depth is buying candidates that fusion cannot meaningfully rank. That is the empirical stopping signal, and it usually arrives well before latency becomes the binding constraint. ## Costs of depth - **ANN latency.** Retrieving more neighbours means exploring more of the graph or scanning more clusters; the cost grows with depth, gently at first and then not. - **Lexical scan work.** Deeper top-k means maintaining a larger heap over longer posting lists. - **Deduplication.** The two lists overlap and may key documents differently — chunk id versus document id — so the merge has to canonicalize. Deeper lists make identity bugs more visible and more expensive. - **Downstream payload.** Everything after fusion pays for a longer list. Because both legs are usually issued in parallel, the added latency is the slower leg's increase, not the sum — which is why moderate depth is cheap and why shallow per-source budgets are rarely justified on latency grounds alone. ## The defensible answer Fetch several times the final list size from each leg, set the depth from per-leg recall curves rather than from the final k, allow the two budgets to differ, and re-check the numbers whenever the corpus grows substantially — recall@50 on a 50k-chunk index is not recall@50 on a 5M-chunk index.