Your LlamaIndex query engine misses its p95 latency budget — which knobs do you trade?

## Attribute before you tune The failure mode in latency work is guessing. A LlamaIndex query engine has three stages and each can dominate, so measure them apart: - **Retrieval**: time `retriever.retrieve(q)` directly. This is one embedding call plus a vector-store round trip — typically tens of milliseconds. - **Postprocessing**: the difference between the retriever's time and the engine's time, minus synthesis. In practice you isolate it by running the engine with and without `node_postprocessors`. - **Synthesis**: everything left. Sanity-check it by counting LLM calls — the response mode determines that count, and it is the number people most often get wrong. A useful trick during attribution is a `no_text` response mode, which runs retrieval and postprocessing and skips generation entirely, giving you the front half's cost cleanly. ## The knobs, and what each one actually costs **`similarity_top_k`.** Almost free in the vector store, expensive everywhere downstream. It is the multiplier on reranker work and, without a reranker, on prompt size. Cutting it is the fastest win and the one that most directly costs recall. Never cut it blindly — check the recall@k curve first, because if the curve flattened at 15 and you are running 50, the first 35 are pure latency. **Reranker choice and placement.** A cross-encoder does one forward pass per candidate, so its cost is `similarity_top_k` times per-pair inference. Options in descending quality: hosted reranker (adds a network hop and a rate limit), local cross-encoder (adds CPU or GPU contention), smaller cross-encoder, no reranker. Reranking is also the most natural thing to shed under load, because the pipeline still returns an answer without it. **Response mode.** The dominant synthesis term. `refine` issues one sequential LLM call per node — ten nodes is ten round trips, and that alone will miss most interactive budgets. `compact` usually collapses to a single call. `tree_summarize` makes more calls but its batches at each level are independent, so `use_async=True` turns node count into tree depth. Check what mode you are actually running before optimizing anything else; a mode set months ago for a summarization endpoint and copied into a chat endpoint is a classic finding. **Model choice for synthesis.** A smaller or faster model on the synthesis call often moves p95 more than any retrieval tuning, and the quality loss is bounded when the reranker has already handed it three highly relevant passages. Retrieval quality and synthesis model size are substitutes to a degree. **`streaming=True`.** It does not reduce total latency at all — it changes what the user experiences, moving the perceived measure from full-response time to time-to-first-token. If your SLO is written against total time, streaming does not help you meet it; if it is written against perceived responsiveness, it may be the whole fix. Decide which SLO you are actually defending. ## Budget per stage, then defend it Write the budget down as a split, for example: 50 ms retrieval, 150 ms rerank, 800 ms synthesis, with 2 s p95 end to end. A per-stage budget makes regressions attributable — when p95 moves, one line moved — and it makes the tradeoff conversation concrete, because raising top-k is now visibly spending someone else's milliseconds. Then decide the degradation path in advance. Under pressure a query engine can shed work gracefully: drop the reranker and pass fewer retrieved nodes straight through; fall back to a cheaper synthesis model; cap `similarity_top_k`. Each is a quality reduction that still returns an answer, which is almost always better than a timeout. Choose the order deliberately and make it observable so a degraded response is visible in traces rather than a mystery. ## The things that are not latency problems Two confounders masquerade as engine slowness. First, the hosted dependencies — embedding endpoint, reranker API, LLM provider — have their own tail behaviour, and a p95 that is fine at the median and terrible at the tail often reflects a provider's tail rather than your configuration; timeouts and retries there need explicit policies, and a naive retry doubles the worst case. Second, cold state: a local cross-encoder loading its model, or an in-memory lexical index rebuilding at boot, makes the first requests after a deploy dramatically slower. Warm those on startup rather than accepting a latency spike on every rollout. ## What good judgment looks like here The principal-level answer is not a list of knobs; it is the discipline of attributing before tuning, holding one knob fixed while moving another, expressing the target as a per-stage budget, and pre-deciding what quality you will trade when the budget is breached in production rather than improvising during an incident.