Multi-Query and Decomposition

## The problem both techniques attack A vector retriever searches with whatever the user typed. That text is embedded into a single vector, and the index returns the chunks whose embeddings are nearest to it. Two things routinely go wrong. First, the user's phrasing may simply not resemble the phrasing of the passage that answers it, so the right chunk is not in the top-k. Second, the user may have asked several things at once, and no single passage in the corpus answers all of them — the top-k fills with documents that match the loudest term in the question while the other parts go unretrieved. Multi-query fan-out addresses the first problem. Decomposition addresses the second. They are frequently confused because both start by calling an LLM on the query and both end with more than one retrieval, but the shapes differ in a way that determines your execution plan. ## Multi-query fan-out You prompt a model: "generate four alternative phrasings of this question." Each phrasing is embedded and searched independently, and the result lists are merged into one candidate set for the generator. Nothing about paraphrase two depends on paraphrase one, so all searches run in parallel and the wall-clock cost is one LLM call plus one round of concurrent retrievals. The intuition is coverage. An embedding is a point; the region of the corpus you actually want may be an area. Several paraphrases scattered around the original give you several nearby points, and their union covers more of that area. It is most valuable when the input is short and idiosyncratic — a two-word bug-tracker title, a search box query, an internal abbreviation — because short text embeds noisily and small wording changes move the vector a long way. Note that the *answer* is one answer; you are not splitting the information need, only the way you ask for it. ## Sub-question decomposition Here you prompt a model to split the information need itself: "list the sub-questions that must be answered to answer this." The output is a small plan. For "how did our EMEA gross margin change after the 2023 acquisition?" a reasonable plan is (1) what was EMEA gross margin in the periods before the acquisition, (2) what was it after, (3) what else changed in the segment definition. Each is retrieved for separately — likely hitting different 10-K filings and different sections — and each gets its own short answer. A final call joins those sub-answers into the response. Decomposition changes the *unit of retrieval*. Instead of hoping one top-k contains evidence for every part, you give every part its own budget of chunks. That is why it rescues compound questions that fan-out cannot: paraphrasing a two-part question just produces four two-part questions, all of which retrieve the same lopsided results. ## Independent versus dependent sub-questions A decomposition plan is a small dependency graph, not always a list. Some sub-questions are independent and can be retrieved concurrently. Others are dependent: you cannot ask "what is that carrier's claims SLA?" until you know which carrier serves the lane. Dependent chains serialize, so latency grows with depth rather than breadth, and each hop compounds the risk that a wrong intermediate answer sends the next retrieval somewhere useless. Where the sub-questions are independent, keep them independent — it is cheaper and more robust. ## Choosing between them Useful heuristics: - Do the parts of the question have *different* answers found in *different* documents? Decompose. - Is the question one intent whose vocabulary probably differs from the corpus? Fan out. - Is the question a single lookup already phrased in corpus vocabulary? Do neither — plain retrieval is fine, and transformation only adds latency. - Is the question compound *and* oddly worded? Decompose first, then optionally fan out each sub-question. Do not do both by default; the call count multiplies. ## What it costs and how it fails Both techniques trade calls for recall. Fan-out adds one planning call plus N retrievals. Decomposition adds a planning call, N retrievals, usually N answering calls, and a join call — an easy 5x on both tokens and latency for a three-part question. The characteristic failure of fan-out is *drift*: the model rewrites the question into paraphrases that quietly change its meaning, pulling in confidently-retrieved but irrelevant chunks that then outvote the correct ones. Constrain the rewriter to preserve entities, numbers and dates. The characteristic failure of decomposition is a *bad split*: the plan omits a part, invents a part the user never asked about, or splits along a seam the corpus does not follow, so each sub-retrieval is individually plausible and the joined answer is subtly wrong. Log the generated sub-questions — when a decomposition-based system gives a strange answer, the plan is usually where it went wrong, and it is the cheapest artifact to inspect.