How do you decide whether hybrid retrieval is worth its cost over keyword search alone?

## Frame it as a segment question The unproductive version of this debate is "is semantic search better than keyword search". The productive version is: *which queries does the current system fail, how many of them are there, and would a second retriever fix those specific failures?* Segment the query log: - **Navigational / known-item.** The user knows what they want and types an identifier, an exact title, a part number, an error code. Keyword retrieval is close to ideal here, and a dense leg contributes mostly noise — embeddings are famously weak on rare tokens and exact strings. In many catalogues, support portals and internal tools this is the majority of traffic. - **Descriptive / natural language.** Symptoms, questions, paraphrases, "the thing that does X". Exactly where term matching fails and dense retrieval earns its keep. - **Zero-result and reformulation tails.** Queries returning nothing, or followed within seconds by a rewritten query, or abandoned. This is your measured evidence of failure and the cleanest estimate of the addressable segment. - **Cross-lingual and multilingual.** Users querying in one language against documents in another. Lexical retrieval cannot bridge this at all without translation infrastructure; a multilingual encoder can. If the addressable segment is 25% of traffic, the case makes itself. If it is 2%, hybrid is a large permanent cost to move a small number, and cheaper interventions may match it: a curated synonym list, better field weighting, fixing the analyzer for your domain's tokenization, adding an autocomplete or query-suggestion layer, or mining click logs for query rewrites. Those are unglamorous and frequently win on effort-per-point. ## Price the whole system The cost of hybrid is not "one extra query". It is a permanent second pipeline: **Ingest.** Every document must be embedded, which means model serving capacity at ingest rate, and a re-embedding pass over the entire corpus whenever the model changes. For a large or fast-changing corpus this dominates. Ask directly: how long does a full re-embed take, what does it cost, and can we serve traffic during it? **Memory.** Vector indexes are memory-resident structures whose footprint scales with document count and dimension, plus overhead for the index structure itself. This is often the line item that surprises teams and the reason quantization enters the conversation. **Latency.** Two retrievals plus fusion, and the query must be embedded first — which puts a model inference on the critical path of every search. The p99 is worse than the sum of the medians because you wait for the slower leg. If your latency budget is tight, this alone can decide it. **Tuning and evaluation.** Fusion method, blend weights or fusion constant, per-leg retrieval depth, filter strategy. Each is a parameter someone must own, tuned against a judgment set that must exist and be maintained. Without an evaluation programme you cannot tell an improvement from a regression, and the honest answer is that hybrid without evaluation is a coin flip. **Model lifecycle.** An encoder is a dependency with a version, a deprecation schedule if hosted, and drift relative to your corpus. Every upgrade invalidates thresholds and blend weights and requires a full re-embed. This is a multi-year commitment, not a project. ## Prove it offline first Build a judgment set from the addressable segment — real queries, sampled from logs, with graded relevance labels — and evaluate the current system, a dense-only system and the fused system on it. Weight the metric to what the product cares about: precision at small k for a search box people scan, recall at larger k when a reranking or generation stage follows. Look at the per-query deltas, not just the average: the number one risk of a relevance change is not that the mean moves little, it is that it wrecks the queries that used to work. A change that lifts the mean while destroying a hundred head queries will get reverted the week it ships. Also test the regression case explicitly: identifier and exact-title queries. If the dense leg drags a perfect keyword hit off the first position, you have found the failure mode most likely to generate complaints. ## Roll out by routing Hybrid is not all-or-nothing per system; it is per query. A cheap classifier or even a set of heuristics can route: queries that look like identifiers or exact titles go keyword-only; long, descriptive, question-shaped queries get both legs. This makes the cost proportional to the benefit, protects head traffic from ranking regressions, and gives a natural staged rollout — one query class at a time, measured online, easy to unwind. A second, cheaper posture is **fallback**: run keyword alone, and invoke the dense leg only when the keyword result is empty or weak. Latency stays at baseline for most traffic, cost is confined to failures, and the change is trivially reversible. It captures much of the zero-result recovery without paying on every query — though it does not help queries that return plenty of mediocre results. ## What to say when the answer is no Be prepared to argue against building it. "Our traffic is 80% part numbers, our corpus is 40,000 documents, our latency budget is 100ms, and we have no labelled data" is a complete and defensible case for spending the same effort on analyzer quality, synonyms and query suggestions instead. The principal-level skill is not knowing how to build hybrid retrieval; it is knowing which of these facts to establish before anyone starts.