In late chunking, why is the whole document encoded before chunk vectors are pooled?

## What a chunk vector normally encodes The conventional pipeline is split-then-embed. A document is cut into chunks, each chunk is sent to an embedding model on its own, and the resulting vector is stored. The encoder's attention never crosses the chunk boundary, so whatever the chunk fails to say explicitly is simply absent from its vector. That is fine for self-contained prose and disastrous for the way real documents are written. Human writing is full of anaphora and elision: "it", "the programme", "this policy", "the above", "the same limit applies". A chunk deep in a report that reads "It was suspended in March after the second audit" is, standing alone, about suspension in general. A query naming the actual programme will not land near it, because the programme's name appears only on page one. ## What late chunking does differently Late chunking needs a long-context embedding model — one that exposes per-token output and accepts a window large enough to swallow a whole document, with roughly 8k tokens being a common ceiling. The pipeline becomes: 1. Feed the entire document through the encoder in one pass. Every token now has a contextual embedding computed with attention over the whole document. 2. Apply the chunk boundaries to the token sequence. 3. Mean-pool each chunk's token span into a single vector. 4. Store those vectors against the same chunk texts as before. Boundaries are applied after encoding rather than before it — that is the entire trick, and it is where the name comes from. The chunk that says "It was suspended in March" now pools tokens whose representations were computed while the model could see the programme's name, so the vector sits near queries naming it. ## What it does not change This is the distinction candidates most often miss. Late chunking changes the *vector*, not the *text*. You retrieve the same chunk string you always did, so the model still reads "It was suspended in March" with no antecedent. Late chunking improves the probability the right chunk is found; it does nothing on its own for the model's ability to interpret it once retrieved. Systems that need both pair late chunking with some read-time expansion or with metadata in the payload. It also does not change index shape, granularity, chunk count, storage cost, or query-time latency. One chunk still yields one vector. Nothing downstream — filtering, hybrid scoring, reranking — needs to know late chunking happened. ## Compared with rewriting the chunk text There is a competing family of techniques that generates a short document-situating summary and prepends it to each chunk before embedding. Both approaches attack the same failure, from opposite ends: one enriches the encoding, the other enriches the text. The economics differ sharply. Prepending costs a generation call per chunk at index time and inflates the stored text, but it survives any embedding model and can inject knowledge from outside the document. Late chunking costs one encoder pass per document, adds no generated tokens, and is deterministic — but it needs a long-context encoder and it can only redistribute context that already exists inside the document. ## Constraints worth naming in an interview **Document must fit the encoder window.** Longer documents force overlapping macro-windows, and the boundary between windows reintroduces exactly the context loss you were removing. **Pooling is lossy.** One vector per chunk is still one vector. A very long chunk pools many tokens into an average, which blurs; the technique does not rescue bad chunk sizing. **Pipeline requirements.** You need token-level embeddings and your own pooling step. Not every hosted embedding endpoint returns token vectors — many return only the pooled sentence vector, which forecloses the technique entirely. **Benefit is uneven.** Gains concentrate on terse, anaphora-heavy material — meeting minutes, tables, legal clauses, numbered procedures. Self-contained chunks that already repeat their subject gain almost nothing, which is why a blanket rollout can look disappointing in aggregate while being decisive on a specific slice. ## How to evaluate it Hold the chunks, the queries and the retrieval mechanics fixed; swap only the vectors. Compare retrieval recall at your operating k for independently embedded versus late-chunked vectors, then slice the results: expect the gain to sit almost entirely on queries whose answer chunk never contains the query's key entity string. Check the same slice against lexical retrieval, because a hybrid pipeline may already be catching those cases by exact term match, in which case late chunking is buying you less than the headline number suggests.