In RAG, why embed small child chunks but return their larger parent sections?

## The tension this resolves Every chunking discussion eventually hits the same wall. Make chunks small and each embedding represents one clear idea, so the nearest-neighbour search is precise — but the retrieved text may be a fragment that mentions the answer without explaining it. Make chunks large and there is enough surrounding argument for the model to work with — but the single vector now has to represent five subjects at once, and it sits in a mushy region of the embedding space where it matches everything weakly and nothing strongly. The insight of small-to-big is that these are two different jobs being forced onto one object. Searching wants a specific, narrow unit. Reading wants a complete, self-contained unit. There is no reason they have to be the same text. ## The mechanism At index time: split the document into parents — typically a subsection, a heading-bounded block, or a few paragraphs. Subdivide each parent into children — a sentence, a couple of sentences, a small paragraph. Embed the children only. Store each child vector with a `parent_id` pointing at the parent's text in a document store (which can be a plain key-value store; it never needs to be a vector index). At query time: embed the query, retrieve top-k children, collect their parent ids, deduplicate, fetch those parents, and pass the parent text to the generator. The vector index therefore holds many small, sharp vectors; the generator sees few large, complete passages. Note that this changes how much text you send, so how those enlarged passages get ordered and packed into the prompt is a separate concern from the retrieval design here. ## Sentence-window expansion: the same idea, cheaper A lighter variant does the expansion at read time instead of maintaining an explicit hierarchy. Chunk into individual sentences, embed each, but store with each sentence its position in the document. On a hit, return that sentence plus the *k* sentences before and after it. There is no parent object to define, no hierarchy to maintain, and one parameter to tune. Sentence-window is usually enough when the corpus is flowing prose and the answer's supporting context is genuinely adjacent — transcripts, articles, narrative reports. Explicit parent-document retrieval is better when the meaningful unit has real boundaries you would rather not cross, such as a documented API method or a clause in a contract, because a fixed window will happily run off the end of one and into the next. ## Where it costs you Context budget. Ten child hits can expand into ten parents. If parents are large, you have just quadrupled the tokens sent per query, with the cost and latency that implies — and pushed genuinely relevant text into the middle of a long prompt, where models attend to it less reliably. Deduplication is mandatory. Three children from the same parent must yield one copy of that parent, not three. Skip this and you burn budget on duplicates and skew any downstream reranking. Parent granularity is the real knob. Too small and you have not solved anything; too large — a whole chapter — and you are back to stuffing loosely relevant text at the model. A subsection-sized parent is the usual sweet spot. Ranking becomes indirect. Your top-ranked child may belong to a parent that, read whole, is less useful than the parent of the third-ranked child. If several children of one parent are hit, that is a reasonable signal to promote it. ## When it is the wrong tool If the child chunk already answers the question completely — short FAQ entries, product records, log lines — expansion adds tokens and no information. If your corpus is a set of tiny independent documents, there is no parent to return. And if the whole document is short enough to send anyway, the hierarchy is machinery you do not need. ## How to evaluate it Measure the two halves separately, because they fail differently. For retrieval, check whether the correct child appears in the top-k — that tells you whether the small-vector matching works. For generation, check whether the returned parent contained everything needed to answer — that tells you whether your parent is big enough. A system with high child recall and poor answers has parents that are too tight; a system with good answers but soaring token costs has parents that are too loose. ## The one-line version Embed what makes matching precise; return what makes answering possible. They are allowed to be different objects, and treating them as one is why so many first-attempt RAG systems feel like they retrieve the right thing and still answer badly.