Semantic chunking yields chunks from 40 to 3,000 tokens — what breaks downstream?

## Where the variance comes from A semantic splitter cuts wherever the similarity curve says the topic moved. If a document holds one idea for four pages, that is one chunk; if a speaker fires off three unrelated one-liners, those are three chunks of a sentence each. Nothing in the algorithm bounds the output length, so the size distribution is long-tailed and heavily corpus-dependent. Field ecology notebooks, where entries change site and species mid-paragraph, produce exactly this shape — a mass of short entries plus a few very long uninterrupted observation passages. That is not a bug; coherence was the objective. But every stage after chunking was probably designed assuming roughly uniform pieces, and that mismatch is where production problems appear. ## Failure 1 — oversized chunks and silent truncation Embedding models have a maximum input length. Feed a chunk longer than that and it is typically truncated rather than rejected: the vector then describes the first portion of the chunk while the index stores the whole text. Queries that should match the tail never do, and nobody sees an error. This is the nastiest of the failures because it degrades recall quietly, and it only shows up if you instrument chunk token lengths at ingest. Oversized chunks also distort the prompt. If top-k is 5 and one hit is 3,000 tokens, that single result crowds out four others; on a long-context model it merely costs money, on a tighter budget it means real evidence gets dropped by the packer. Retrieval scores can even look *better* — a huge chunk is more likely to contain the answer somewhere — while answer quality falls because the generator must find a needle in it. ## Failure 2 — undersized chunks with no standalone meaning At the other tail, a 40-token chunk may be a single sentence stranded between two strong breakpoints. Two problems: its embedding is noisy and easily matched by superficially similar queries, and even when it is the *right* passage it lacks the surrounding sentences that make it interpretable. "That figure was revised down after the site visit" retrieves cleanly and answers nothing. Short chunks are also over-represented in results by some scoring schemes, because a short text about exactly one thing has a tight, well-aligned vector. ## Failure 3 — unpredictable capacity per query With a fixed k, evidence volume becomes a lottery. One query returns 400 tokens, another 9,000. That propagates into token cost per query, time-to-first-token, and the risk of overflowing the model's window on the tail queries — the ones you notice last, in production, on the users who ask the broadest questions. ## The fixes **Bound the distribution after splitting.** Post-process the semantic output with two guardrails: merge any chunk below a minimum token count into its neighbour (the neighbour on the side with the smaller boundary distance is the sensible choice), and re-split any chunk above a maximum, falling back to a cheaper length-based split inside it. This keeps the semantic boundaries you found while clipping both tails. It is a couple of dozen lines and it is the single highest-value addition to a naive semantic pipeline. **Never exceed the embedder's input limit.** Make the maximum chunk size strictly smaller than the embedding model's limit, and assert it at ingest rather than trusting the splitter. Log the distribution — median, p95, max, and the count of chunks under 100 tokens — as an ingest metric you can alert on when a new corpus arrives. **Pack the prompt by token budget, not by k.** Retrieve generously, then fill the context up to a token ceiling in rank order, stopping when the next chunk would not fit. This converts unpredictable evidence volume into a fixed budget and makes cost per query stable. **Consider a reranker.** With variable sizes, first-stage vector scores are noisier, and a cross-encoder rerank over a larger candidate set recovers much of the ordering quality before the packing step. **Score short chunks with care.** If tiny chunks keep surfacing and disappointing, either raise the minimum size or attach a little neighbouring context at retrieval time so what reaches the model is interpretable. ## What good looks like in an answer The strong version of this answer names the *silent* failure first — truncation at the embedder — because it is the one that does not announce itself, then gives the guardrail pair (min-merge, max-resplit) and the budget-based packing change. The weak version says only "chunks should be a consistent size", which misses that consistency was traded away deliberately and the job is to bound the tails, not to abandon the method.