In LlamaIndex, how do the compact, refine and tree_summarize response modes differ?

## Where the mode sits A LlamaIndex query engine has two halves: retrieval, which produces nodes, and synthesis, which produces prose. The response mode configures the second half. You set it either inline — `index.as_query_engine(response_mode="compact")` — or explicitly with `get_response_synthesizer(response_mode=ResponseMode.TREE_SUMMARIZE)` and hand the result to a `RetrieverQueryEngine`. The choice does not change *what* was retrieved; it changes how many LLM calls happen, in what order, and how much of each node the model ever sees. ## refine The original strategy. The first node goes into a QA prompt (`text_qa_template`) and yields a draft answer. Each subsequent node goes into a refine prompt (`refine_template`) together with the current answer, and the model is told to improve the answer using the new context, or to repeat it unchanged if the new context adds nothing. Properties: every node gets its own dedicated prompt, so nothing is truncated away by packing. Cost is N calls for N nodes, strictly sequential — call k+1 needs the output of call k, so there is no parallelism to exploit. With `similarity_top_k=10` you pay ten round trips per query. It is also drift-prone: weak models sometimes discard good material during a refine step, or ramble as the answer is rewritten repeatedly. ## compact (the default) Compact is "compact and refine". Before calling anything, it concatenates node texts into the largest chunks that still fit the prompt window, producing a small number of packed contexts. It then runs the refine chain over *those*, not over individual nodes. In practice most top-k retrievals fit into one packed prompt, so compact is a single LLM call where refine would have been ten. That is why it is the library default: same templates, same refine semantics on overflow, a fraction of the latency and cost. Its only real weakness is the flip side of packing — several passages share one prompt, so the model's attention is divided across them, and a strong passage can be crowded by neighbours in the same context block. ## tree_summarize A different shape entirely. Nodes are grouped into batches sized to the context window and each batch is summarized against the query using `summary_template`. The resulting summaries are then themselves batched and summarized, and so on until a single text remains — a bottom-up reduction tree. What this buys: every node influences the final answer through its level of the tree, so no passage is simply refined away, and the mode scales to node counts that would make refine absurd. It also parallelizes — sibling batches at a level are independent, and `use_async=True` runs them concurrently, so wall-clock time grows with tree *depth* (logarithmic) rather than node count. What it costs: more total LLM calls than compact for the same nodes, and information loss at each level, because a summary of a summary is lossy. For a pointed factual question ("what is the refund window?") that lossiness is pure downside — the answer was in one node and compact would have quoted it. For "summarize the risks across these forty filings", it is exactly right. ## Choosing - **Pointed question, small top-k, latency matters** → compact. This covers most chat-style RAG. - **Pointed question, and you suspect packing is diluting a key passage** → refine, accepting N calls; or better, keep compact and fix precision with a reranker so fewer, better nodes are packed. - **Question genuinely spans the whole retrieved set — summaries, comparisons, exhaustive extraction** → tree_summarize with `use_async=True`. Other modes exist for narrower jobs: `simple_summarize` merges and truncates everything into a single call (cheapest, silently drops text), `accumulate` and `compact_accumulate` apply the query to each chunk separately and concatenate the results (good for per-document extraction, not for prose), `no_text` runs retrieval and skips generation entirely (useful for evaluating retrieval or inspecting `source_nodes`), and `generation` ignores context and just asks the LLM. ## Cost model in one line Compact ≈ ceil(total context tokens / window) calls, mostly one. Refine = one call per node, sequential. Tree_summarize ≈ one call per batch per level, parallelizable. Multiply by your per-query rate to see why the mode, not the model, is often the dominant term in a RAG bill — and why quietly switching modes to "improve answers" can double latency without anyone noticing.