When does a recursive summary tree beat flat parent-child chunking in RAG?

## What the construction is Start with ordinary leaf chunks. Embed them. Cluster the embeddings into groups of related chunks — note that clusters are formed by similarity, so they can span sections that are far apart in the document, which is the point. Summarize each cluster with a model into a single node. Embed those summary nodes and repeat: cluster the summaries, summarize the clusters, and continue until you reach a small number of top-level nodes. The result is a tree whose leaves are verbatim text and whose internal nodes are generated abstractions of increasing scope. Crucially, every node at every level goes into the same index. A query is matched against leaves and summaries together, so the level of abstraction that gets retrieved is decided by the query rather than fixed by the designer. ## The failure mode it targets Flat retrieval, including the small-to-big variant, answers "which passage is most similar to this query?" That is the right question when the answer lives in one place. It is the wrong question for synthesis. Take a 700-page maintenance manual and ask how the inspection regime evolved, or which procedures share a common safety precondition, or what the general posture is on deferred defects. The answer is distributed across dozens of passages, none of which is individually a strong match, and each of which contributes a fraction. Increasing k does not rescue this — it returns more fragments and pushes the model toward summarizing a pile rather than answering. A summary node has already done that aggregation once, at build time, with the whole cluster in view. Retrieving it gives the model a synthesized statement plus, if you want, the leaves beneath it. ## What it costs Build cost is the visible one: a model call per internal node, and the tree has many. It is index-time and amortised, but unlike per-chunk prefixing it also has a *depth* multiplier — you generate at every level. Maintenance is the underrated one. When a leaf changes, every ancestor summary above it is now stale. Correctly propagating updates means re-summarizing a path to the root, and if clustering itself shifts, the tree's shape changes. Many teams end up rebuilding periodically rather than incrementally, which sets a hard bound on freshness. For a manual revised quarterly that is fine; for a corpus revised hourly it is disqualifying. Fidelity loss is the dangerous one. Summaries drop specifics — torque values, part numbers, dates, thresholds — which are exactly the details a technical user needs. A system that answers a broad question well and a precise one vaguely has often over-retrieved from the summary levels. Provenance is the governance one. A retrieved summary is model-generated text. Citing it cites your own pipeline, not the source. In regulated domains you generally must resolve every summary back to the leaves it covers and cite those, which means storing the child links and doing the resolution before showing anything to a user. ## The judgement call There is no consensus that hierarchical trees are worth it in general, and it is fair to say so in an interview. What is defensible is a decision procedure. First, characterize the traffic. Sample real questions and label each as local (answerable from one passage) or cross-cutting (requires aggregation across many). If cross-cutting is a few percent, a tree is a large permanent cost for a small slice, and query decomposition or a broader-context model may serve that slice more cheaply. Second, try the cheap approximations before the tree. Per-document summaries indexed alongside the chunks capture a lot of the benefit at a fraction of the build and maintenance cost. Metadata aggregation, or letting the system fetch a whole document when several of its chunks hit, covers more. Third, if you do build it, decide the retrieval policy deliberately. Matching all levels at once is simple and works well; alternatively route by query type, sending broad questions to summary levels and specific ones to leaves. And set a rule for when a retrieved summary must be expanded into its leaves. Fourth, budget the rebuild. Decide the refresh cadence up front and confirm the corpus tolerates that staleness, because retrofitting incremental updates onto a clustering-based tree is genuinely hard. ## How you would know it earned its place Split your eval set by question type. The tree should show a clear win on the cross-cutting slice and no regression on the local slice — regression there is the tell that summaries are outranking the leaves that hold the specifics. Track build cost and rebuild latency as first-class metrics, not footnotes, because they are what you will actually be managing a year later.