How do you build hybrid BM25-plus-vector retrieval in LlamaIndex?

## Why hybrid at all Dense vector retrieval matches meaning and is weak precisely where users are most literal: SKU codes, error identifiers, function names, rare proper nouns, version strings. Those tokens carry little semantic signal, so their embeddings sit near everything and nothing. Lexical scoring — BM25 — is the opposite: it nails exact and rare terms and is blind to paraphrase. Hybrid retrieval runs both and merges, so the recall floor is the union rather than either leg alone. In LlamaIndex there are two ways to assemble it. ## Route 1: native hybrid in the vector store Several vector-store integrations implement hybrid search themselves, keeping a sparse or keyword index alongside the dense one. In that case you ask the retriever for it: `index.as_retriever(vector_store_query_mode="hybrid", similarity_top_k=10)` The store executes both legs and returns one merged, ranked list. Some integrations expose a weighting parameter — commonly `alpha` — but the meaning and even the direction of that parameter is defined by the store, not by LlamaIndex, so verify it against the specific integration rather than porting a value from another product. Advantages: one network round trip, one system of record, and the keyword index stays consistent with writes because the store owns both. Disadvantage: you inherit whatever fusion the store implements, and you can only use it if your store supports it. ## Route 2: framework-side fusion When the store has no hybrid mode, or when you want control over the merge, build the two retrievers separately and fuse them. `BM25Retriever` lives in the `llama-index-retrievers-bm25` package and is constructed with `BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)` or from a docstore, `from_defaults(docstore=index.docstore, similarity_top_k=10)`. It tokenizes the corpus and builds a BM25 index **in memory, at construction time**. `QueryFusionRetriever` takes the list of retrievers and merges their outputs. The parameters that matter: - `mode` — `"reciprocal_rerank"` for reciprocal rank fusion, which combines by *rank position* rather than raw score. This is the safe default precisely because BM25 scores and cosine similarities are on incomparable scales; RRF never has to compare them. Score-based modes exist (relative-score and distance-based normalization) for when you want magnitude to matter. - `similarity_top_k` — how many fused results the retriever finally emits. Each child retriever keeps its own `similarity_top_k` for the candidates it contributes; setting only the outer one starves the fusion. - `num_queries` — by default the fusion retriever asks an LLM to generate several query variants and retrieves for each. That is query expansion bundled into the retriever, and it adds an LLM call plus more store round trips to every retrieval. Set `num_queries=1` to turn it off when you only want fusion. - `use_async` — issue the child retrievals concurrently, which matters once you have two legs times several generated queries. ## The operational trap in BM25Retriever This is the part interviewers probe. The vector store is a live service: you write new nodes and the next query sees them. `BM25Retriever` is a process-local structure built from the nodes you handed it. Consequences: - **Staleness.** Documents added after construction are invisible to the lexical leg. Your hybrid pipeline silently degrades to vector-only for new content, and nobody gets an error. - **Memory and startup cost.** The corpus text lives in the application process, and building the index costs time on every boot. That is fine for tens of thousands of nodes and untenable for tens of millions. - **Replica skew.** Each service replica builds its own copy, so two replicas can disagree about lexical results if they started at different times. Mitigations: rebuild on a schedule tied to ingestion, persist and reload the built retriever so boots are fast, or move the lexical leg into a store that supports native hybrid so consistency is someone else's problem. At real scale the last option is usually the answer. ## Evaluating whether it helped Hybrid is not automatically better; it costs a second retrieval path and, with query generation on, an LLM call. Measure it: take a labelled question set, compute recall@k for the vector leg alone, the BM25 leg alone, and the fusion. If the union recall is not meaningfully above the vector leg, the corpus is not lexical enough to justify the moving parts. Where hybrid does win, the win is usually concentrated in a query class — identifiers, codes, names — which is worth knowing because it tells you whether a cheaper fix (better chunk metadata, a metadata filter, an exact-match prefilter) would do. Finally, hybrid and reranking are complements, not alternatives: fusion widens the candidate pool across two retrieval philosophies, and a cross-encoder reranker then cuts it down to the few nodes the LLM should actually read.