Why does hybrid retrieval add BM25 when dense embeddings already work?
answer
- two retrievers, two different failure modes
- embeddings summarize meaning, not characters
- rare codes blur into similar codes
- exact identifiers need literal term matching
- measure per query shape, not averaged
basics
~20 sDense embeddings match meaning, so they blur rare literal tokens like part numbers and error codes into similar-looking neighbours. Lexical search such as BM25 matches those tokens exactly. Hybrid retrieval runs both so paraphrased questions and exact-identifier lookups both work.
solid answer
~50 sThe two retrievers fail in opposite directions, and that is the whole argument for running both. A dense bi-encoder compresses a chunk into one vector, so it is strong on paraphrase and synonymy — "belt keeps slipping" finds a passage about drive-tension loss — but weak on rare literal strings. A code like `ERR_1042` or a spec like `M8x1.25` is shattered into subword pieces, is rare or absent in the embedding model's training data, and ends up close to *other* codes rather than to itself. BM25 has no idea what the string means, but it scores documents on exact term overlap with IDF weighting and term-frequency saturation, so a rare token is exactly the case it handles best. In a manufacturing support index full of part numbers, the two legs recover different documents for different query shapes, which is why you diagnose complementarity per query type rather than by a single average score.
go deeper
Be able to say plainly that embeddings match meaning while BM25 matches exact words, and give one concrete example of each winning — a paraphrased question versus a part number.
Explain the mechanism behind the failure: subword tokenization, rare terms unseen in training, and one vector summarizing a whole chunk. Also explain BM25's vocabulary-mismatch weakness in the other direction.
Show that you evaluate complementarity per query slice rather than on a blended average, and that you can decide from data whether the lexical leg earns its operational cost on a given corpus.
Own the tradeoff of running and syncing two indexes across the whole platform: which corpora justify it, how query mix shifts over time, and how you keep the decision revisitable rather than a permanent default.
## Two different definitions of "match" Retrieval systems answer the question "which documents are relevant to this query?", but dense and lexical retrievers define relevance in incompatible ways. A **dense retriever** runs the query and each document chunk through an embedding model — a neural network that maps text to a fixed-length vector (a list of a few hundred to a few thousand numbers). Similar meanings land near each other, so retrieval becomes nearest-neighbour search under cosine similarity or dot product. Nothing in that pipeline preserves the literal characters of the input; the vector is a lossy summary of meaning. A **lexical (sparse) retriever** such as BM25 does the opposite. It builds an inverted index from term to document list and scores a document by how many query terms it contains, weighted by how rare each term is across the corpus (inverse document frequency), with diminishing returns as a term repeats and a penalty for document length. It has no notion of meaning at all, only of literal token overlap. ## Where dense retrieval fails Rare, high-information identifiers are the classic hole. Take a manufacturing support corpus: fault codes (`ERR_1042`), fastener specs (`M8x1.25 bolt`), firmware revisions, SKUs. Three things go wrong at once: 1. **Tokenization shatters them.** Subword tokenizers split `ERR_1042` into fragments that carry no distinctive signal, so the pieces contribute little to the final vector. 2. **They are out of distribution.** The embedding model rarely saw these strings during training, so it has no learned representation to place them precisely. 3. **One vector per chunk.** A chunk mentioning a code once, among 400 other words, has that mention averaged away. The chunk's vector reflects its dominant topic, not the identifier. The result is not a miss with an obvious symptom — it is a *plausible* miss. The nearest neighbours of `ERR_1042` are passages about other error codes, which look relevant to a casual eyeball and are wrong. ## Where lexical retrieval fails BM25's weakness is vocabulary mismatch. If the user writes "the conveyor keeps jamming" and the manual says "transport belt seizure", there is no term overlap and the score is near zero. It also cannot handle a question whose answer is phrased entirely differently, and it is brittle to morphology and spelling beyond whatever stemming the analyzer does. Any query expressed in the user's words rather than the corpus's words is where dense retrieval earns its keep. ## Hybrid retrieval as the answer Hybrid retrieval issues the query to both legs and merges the two ranked lists into one. The premise is not that one leg is better — it is that their *errors are uncorrelated*. When two retrievers fail on different queries, the union of their top results has materially higher recall than either alone, and the merge step decides the final ordering. ## Diagnose per query type, not on the average The common analysis mistake is to compute one average metric over a mixed evaluation set. If 85% of your traffic is prose questions where dense already wins, the lexical leg's contribution is diluted into the noise and the average barely moves — while the 15% of queries that are identifier lookups go from unusable to reliable. Those are often the highest-stakes queries in the system: a technician typing a fault code wants the one page about that code, not a page about a similar code. So slice the evaluation. Label queries by shape — natural-language question, exact identifier, mixed jargon phrase, misspelled term — and report recall per slice. That is what tells you whether the lexical leg is carrying anything, and it is also what tells you when the answer is genuinely "dense alone is enough for this corpus", which does happen on clean prose collections with no codes in them. ## What hybrid costs Two indexes to build and keep in sync, two queries per request (usually issued in parallel, so latency is roughly the slower leg plus the merge), and a merge policy to choose and defend. On a corpus with no rare literals and no jargon, that cost buys little. On a technical corpus dense with identifiers, it is usually the single largest retrieval-quality improvement available before touching the model.
- Your hybrid setup shows no average nDCG gain over dense alone. Would you drop the lexical leg?Not on that evidence. Slice the eval set by query shape first. Gains from a lexical leg concentrate on identifier and rare-jargon queries, which are often a small share of traffic and get washed out of an average dominated by prose questions. If those slices improve and they matter to users, keep it; if every slice is flat and the corpus genuinely has no rare literals, then dropping it is defensible and saves an index.
- Why does a strong embedding model still miss a code like ERR_1042 specifically?Three compounding reasons. The tokenizer splits it into subword fragments carrying no distinctive signal; the string is rare or absent in training data, so the model has no learned representation for it; and the chunk's single vector averages one mention among hundreds of other words. The nearest neighbours end up being other, similar-looking codes — a confident wrong answer rather than an empty result.
- Does hybrid retrieval help when the corpus is plain prose with no identifiers?Much less, and sometimes not at all. The lexical leg pays off where literal tokens carry information the embedding cannot encode. On clean narrative prose with consistent vocabulary, dense retrieval already covers the same documents, and the second index mostly buys operational cost. Measure it rather than adopting hybrid reflexively — it is a corpus-dependent decision.
Dense retrieval is a knowledgeable librarian who understands what you meant; lexical retrieval is the index at the back of the book that finds the exact string you typed. You want both on the desk.
saying these in an interview costs you the question
- Says a better embedding model removes any need for lexical search
- Calls BM25 legacy or obsolete rather than complementary
- Judges hybrid only by one average metric over mixed queries
- Thinks dense vector search performs exact keyword matching
- Assumes hybrid beats dense on every individual query