When does semantic chunking not pay for itself compared with cheaper splitting?

## Account for the cost honestly A semantic pipeline pays twice at the embedding layer. Pass one embeds a window for every sentence position in the corpus in order to build the distance curve. Pass two embeds every resulting chunk for the vector index. Sentence-window text overlaps heavily, so pass one is not merely "one extra call per chunk" — it is roughly proportional to sentence count, which on ordinary prose is several times the chunk count. Add the CPU for sentence segmentation, an extra stage in the ingest job, and a threshold that must be tuned and re-tuned, and the true premium is well above the naive 2x on the invoice line. Against that sits a benefit that is real but bounded: chunks that do not straddle topic boundaries, which improves both the retrieval hit rate and the coherence of what reaches the generator. The engineering question is never "is semantic chunking better?" but "is it better *here*, by enough, per unit of cost and complexity?" ## Where it does not pay **The document already carries its own boundaries.** API reference pages, regulatory filings with numbered clauses, structured reports — these publish their seams explicitly. Inferring seams statistically when the author already marked them is strictly worse: it costs more and it is less accurate. Reach for the document's own structure instead. **Documents are short.** If the typical document fits in one or two chunks, there are almost no boundaries to find, and the cost falls entirely on the side with no benefit. Support tickets, product descriptions and short FAQ entries fall here; often the right unit is the whole document. **Ingest is continuous and large.** A one-off historical backfill amortises the double cost across a long-lived index. A pipeline ingesting millions of new documents a day pays it forever, and the same budget spent on a reranker or a better embedding model usually returns more. **Latency at ingest matters.** If freshly written content must be searchable within seconds, an extra full pass over the text plus threshold logic is a real tax on the critical path. **The measurement says no.** The most common honest outcome of a proper sweep — semantic settings against a length-based baseline, scored on a labelled query set — is a gain small enough to sit inside run-to-run noise. That is a legitimate result, and shipping the cheaper pipeline on the strength of it is the senior call. ## Where it clearly does pay Long, unstructured, topically drifting prose with no exploitable markers. A 90-minute city-council meeting transcript that moves from a zoning variance to the library budget with nothing to mark the change. Field ecology notebooks whose entries switch site and species mid-paragraph. Call transcripts, interview recordings, informal notes, chat logs. In these corpora a length-based split reliably produces chunks that straddle two subjects, and the embedding of such a chunk is an average that matches neither query well. That is exactly the deficiency semantic chunking exists to fix. ## How to decide, in practice 1. **Characterise the corpus.** What fraction of documents carry usable structure? What is the length distribution? How often does topic actually drift within a document? 2. **Estimate the recurring cost**, not the one-off — sentences per document times ingest rate, plus the operational cost of a tuned parameter that someone must own. 3. **Run a bake-off on a sample.** Baseline length-based split versus two or three semantic settings, scored end-to-end on realistic queries, with the same retriever and the same k. 4. **Compare against the alternatives for the same money.** A cross-encoder reranker, a stronger embedding model, hybrid keyword-plus-vector retrieval, or query rewriting frequently deliver more improvement per unit of effort than any chunking change. 5. **Segment the decision.** Corpora are rarely homogeneous. Routing structured documents down the cheap path and unstructured ones down the semantic path costs one classifier and beats a global choice. ## The contested part, stated plainly As of mid-2026 the field has no settled verdict that semantic chunking is generally superior. Published comparisons are mixed and highly corpus-dependent, and several report that a well-tuned length split with overlap plus a good reranker matches semantic splitting on many benchmarks. An interviewer asking this question is usually checking whether you will assert a consensus that does not exist. The defensible position: semantic chunking is a targeted fix for a specific deficiency — unstructured prose whose topics drift — and outside that setting it is a cost with an unproven return that you should measure before adopting.