In a RAG pipeline retrieving 8 chunks into an 8k-token context, how do you pick chunk size?

## The arithmetic ceiling The reader's context window sets a hard ceiling on chunk size once you have fixed k. The budget is: context window = system prompt + question + (k x chunk size) + answer headroom. With an 8,000-token window, a typical allocation reserves 500–1,000 tokens for the system prompt and any tool or format instructions, a couple of hundred for the user's question and conversation state, and 800–1,500 for the answer the model must still be able to write. That leaves roughly 6,000 tokens of retrieved material. Divide by k=8 and you get about 750 tokens per chunk. Configure overlap at 15% and the *distinct* content per chunk drops nearer 640 — the duplicated text still consumes context. This arithmetic is why "just use 1,000 tokens" is not an answer. At 1,000 tokens, k=8 alone is 8,000 tokens and the prompt does not fit. The three quantities — chunk size, k and context window — are one budget, and moving any of them moves the others. ## The floor set by the atomic unit The ceiling comes from the reader; the floor comes from the corpus. Every corpus has a smallest unit of text that is *wrong when split*. In warehouse standard operating procedures it is usually a numbered step plus the safety caveat attached to it: "Lower the mast fully" retrieved without "before reversing on a gradient" is not a partial answer, it is a dangerous one. In a policy manual it is a clause plus its exceptions. In a runbook it is a command plus its preconditions. So the sizing procedure is: sample a few hundred of these units, measure their token lengths, take a high percentile — 90th or 95th — and treat that as the minimum viable chunk size. If that floor exceeds the ceiling the context budget allows, you do not have a chunk-size problem; you have a k problem or a context-window problem, and the honest move is to reduce k, move to a larger-context reader, or accept that some queries will need a second retrieval round. ## The precision/context tradeoff between the bounds Inside the feasible band the tradeoff is real and not merely aesthetic. **Smaller chunks** produce embeddings that represent one idea, so similarity against a specific question is sharp and precision is high. You also get more distinct sources in the top-k, which helps questions that need corroboration. The costs are more chunks in the index, more mid-sentence and mid-unit cuts, and answers that arrive without surrounding context — the model sees the rule but not the scope it applies to. **Larger chunks** carry their own context, so the reader has more to work with per hit, and there are fewer boundaries to damage anything. But a single vector must now represent several topics at once, and averaging over more content pulls the embedding toward the document's general subject and away from any specific claim in it. That is dilution: the chunk that contains your answer scores lower because most of it is about something else. Fewer, longer chunks also mean the same k covers fewer independent sources. ## Where length-driven splitting hits its limit Sizing cannot fix everything. A freight tariff table split at any fixed length will eventually cut a rate row so the number lands in one chunk and its unit in another, and no chunk size makes that safe if the table is longer than the window. The diagnostic is worth knowing: retrieval finds the right document, the answer is confidently wrong, and inspecting the retrieved chunk shows a value with no header or unit near it. At that point you have found the boundary of what length-driven splitting can do, and the remedy lives in a different technique rather than in another round of size tuning. ## Measure, do not argue Chunk size is cheap to sweep and expensive to guess. Build a small eval set of real questions with known answer locations, then run the full pipeline at three or four sizes — say 400, 600, 800, 1,200 — holding everything else fixed, and score both retrieval (does the gold passage appear in the top-k?) and end-to-end answer quality. The two often disagree: a size that maximises retrieval recall can lose on answer quality because the chunks are too fragmentary for the reader. End-to-end quality is the metric that matters. ## Signals that the size is wrong - A high fraction of chunks begin or end mid-sentence — the size is fighting the corpus structure. - Retrieved chunks are topically right but never contain the specific fact — chunks are too large, and the answer is being diluted. - The reader keeps saying it lacks information while the right document was retrieved — chunks are too small, or context that lived elsewhere in the document was lost. - Many chunks are far below target — the merge behaviour of the splitter is producing fragments, not the size setting.