Which RAG queries justify multi-query fan-out when the latency budget is one second?

## Account for the latency before arguing about quality Query transformation is usually discussed as a quality technique and paid for as a latency technique. Do the arithmetic before deciding anything. A plain RAG turn is roughly: embed the query (tens of milliseconds), search (single-digit to low tens of milliseconds), generate (the dominant term, hundreds of milliseconds to seconds). Add multi-query fan-out and you insert one model call to produce the paraphrases — sequential, before any retrieval can start — plus N searches that run concurrently and therefore cost about one search's time. So fan-out adds roughly one model round trip. Add sub-question decomposition and the shape is worse: a planning call, then N retrievals, then N answering calls, then a join call. Even with the sub-answers generated concurrently, you have three sequential model stages instead of one. For a live analyst tool this is exactly how a 900ms answer becomes a six-second one — five model calls and five retrievals, most of them on the critical path. Users experience that as a different product. The first conclusion is structural: the extra vector searches are nearly free and the extra model calls are nearly the whole cost. Any optimization that attacks the searches is attacking the wrong term. ## Not every query deserves it The correct default is that most queries do not get transformed. In a typical support or internal-knowledge assistant, the majority of traffic is single-fact lookups already phrased in corpus vocabulary — plain retrieval answers them well, and a transform adds a second of latency for no measurable gain, sometimes for a loss when drifted paraphrases pull in noise. The queries that do justify it share recognizable shapes: - **Compound questions** with multiple clauses, comparisons, or a before/after structure. These need decomposition because no single passage contains all the evidence. - **Terse or jargon-heavy inputs** — a two-word ticket title, an internal abbreviation — where the embedding is noisy and paraphrases materially change what is retrieved. - **High-stakes questions** where a wrong answer costs far more than a slow one: compliance, financial reporting, clinical or legal review. Here seconds are cheap. - **Asynchronous or batch workloads**, where there is no interactive budget at all and you should be more aggressive than in a chat box. ## Gating, not global policy The operational answer is a router. Classify the incoming query cheaply — rules on clause count and question words, a small classifier, or a fast model with a tight output schema — and send only the shapes above through the transform. The classifier must be materially cheaper than what it gates, or you have moved the cost rather than removed it; a rule-based prefilter that catches the obvious single-lookups before any model call is often the highest-leverage piece. Routing also lets you differentiate by *product surface* rather than only by query text. The same engine may back an interactive chat box with a strict budget and a research workflow where a ten-second answer is fine. Those deserve different policies even for identical questions, and hard-coding one global transform pipeline forecloses that. ## Measure the lift per class, not in aggregate Aggregate evaluation hides everything. If fan-out lifts recall substantially on ten percent of queries and does nothing on the rest, the aggregate number looks like a small win and tempts you to apply it everywhere — paying the latency on ninety percent of traffic for nothing. Slice the eval set by the same classes the router uses, and require each class to justify its own transform independently. Log which path each request took so production behaviour can be attributed. Also measure the *distribution*, not the mean. Transformation lengthens the tail more than the median, because a slow planning call and a slow join call compound. A p50 that moves from 900ms to 1.2s may be acceptable while a p95 that moves to eight seconds is not, and only the percentile view reveals that. ## Consider the cheaper levers first Query transformation is one option among several for the same underlying problem, and it is the one that spends the most latency: - **Chunking and metadata.** Many transformation wins are really compensating for chunks that split a rule from its context, or for missing filters. Fixing ingestion costs nothing at query time. - **Hybrid retrieval.** Lexical matching alongside vector search recovers much of what paraphrasing was meant to recover, especially for exact identifiers and rare jargon, and adds negligible latency. - **Larger top-k with a downstream selection stage.** Sometimes retrieving more and choosing better beats searching more times. - **Caching.** Repeated or near-repeated queries can reuse a previously computed plan or result set, removing the planning call entirely on the hot path. A principal-level answer names transformation as a deliberate spend with a measured return on a defined query class — not as a pipeline upgrade applied everywhere because it improved a demo.