In Haystack, how does DocumentJoiner's reciprocal_rank_fusion mode build hybrid retrieval?

## What the joiner is for Hybrid retrieval means running two retrievers over the same corpus and merging their candidate lists. In Haystack that merge is an ordinary component, `DocumentJoiner`, so the whole arrangement stays visible in the pipeline graph and in the serialised YAML. ```python from haystack.components.joiners import DocumentJoiner pipe.add_component("joiner", DocumentJoiner(join_mode="reciprocal_rank_fusion", top_k=10)) pipe.connect("bm25.documents", "joiner.documents") pipe.connect("dense.documents", "joiner.documents") ``` ## The variadic input The joiner's `documents` input is *variadic*: unlike a normal socket, which accepts exactly one connection, it accepts many and delivers them to `run()` as a list of lists. This is the single most-missed mechanical detail. You do not create numbered sockets, and you do not need an intermediate component — you simply connect each producer to the same socket name. The order of those `connect` calls is not cosmetic: it defines the order of the incoming lists, and therefore which entry of `weights` applies to which leg. ## The join modes `join_mode` selects the merge policy: - **`concatenate`** — union of the lists, deduplicated by document id, keeping the highest score seen for a duplicate. Fine when both legs produce comparable scores, which in practice they rarely do. - **`merge`** — a weighted average of the scores of documents present in several lists. - **`reciprocal_rank_fusion`** — scores are discarded and each document is scored from its *rank* in each list. - **`distribution_based_rank_fusion`** — normalises each list using the distribution of its own scores before combining. Rank fusion is the default choice for BM25-plus-dense because BM25 scores are unbounded and corpus-dependent while cosine similarities sit in a narrow band; there is no honest constant that makes them comparable. Working from ranks sidesteps the problem entirely. The practical consequence, and the thing to say out loud in an interview, is that **after RRF the `score` on each Document is a fusion score, not a similarity** — do not threshold it with a cutoff you tuned against cosine values, and do not show it to users as a confidence. ## Deduplication Merging identical documents depends on `Document.id`. Haystack derives that id by hashing the document's content and metadata unless you set it explicitly, so the same chunk written once and retrieved by both legs collapses into one entry with a boosted fused score — which is exactly the behaviour you want, since agreement between two independent retrievers is signal. If your indexing pipeline assigns ids non-deterministically, or writes the same text twice with differing metadata, dedup silently stops working and duplicates survive the join. ## `weights`, `top_k` and `sort_by_score` `weights` takes one number per incoming list and biases the fusion — useful when your corpus is jargon-heavy and the lexical leg deserves more say, or the opposite for conversational queries. Tune it against a labelled query set, not by intuition. `top_k` on the joiner caps the fused output; setting it well below the sum of the legs' `top_k` is the point of the exercise, since the fused list is the candidate pool for a downstream reranker. `sort_by_score` controls whether the output list is ordered by the resulting score; leaving it on is normal, but it matters if a later component depends on incoming order. ## Running the pipeline A hybrid pipeline has two entry points. The BM25 retriever needs `query`, and the dense leg's text embedder needs `text`: ```python pipe.run({"bm25": {"query": q}, "text_embedder": {"text": q}}) ``` Omit one and Haystack will simply not execute that branch, leaving you with a "hybrid" pipeline that is quietly doing single-leg retrieval and mediocre results with no error. A cheap guard is an integration test asserting the joiner received two lists — or, more practically, asserting a known lexical-only match (an exact product code, say) survives to the top of the fused list. ## Where the joiner sits The usual production shape is: two retrievers → `DocumentJoiner` → a similarity ranker → prompt builder → generator. The joiner widens recall cheaply; the ranker restores precision on the widened pool. Fusing and then feeding the raw fused list straight to the generator wastes most of the benefit, because RRF is good at surfacing candidates and mediocre at final ordering.