When is agentic chunking worth its cost for an internal document corpus?

## What the technique is Most chunking is mechanical. You either cut at a fixed token count with some overlap, or you cut on structure the document already carries — headings, clauses, list items, function definitions, log records. Agentic chunking replaces that rule with a judgement: a model reads the document and proposes where each self-contained unit begins and ends, so that a procedure stays whole, a table stays with its caption, and a definition stays with the term it defines. It is attractive for an obvious reason. Boundary errors are a genuine and underdiagnosed source of retrieval failure: a chunk that ends midway through the third step of a five-step procedure will retrieve and mislead. And it is expensive for an equally obvious reason: you have replaced a deterministic function with model inference, once per document, forever. ## The cost side, stated honestly **Inference at index time, repeated.** One call per document, or per window for long ones, at ingest and again on every update. For a corpus of a few hundred guides that is trivial. For a corpus ingesting thousands of items an hour it is a permanent tax with no ceiling. **Nondeterminism.** The same document can yield different boundaries across runs, model versions, or prompt edits. That breaks the property engineers quietly rely on: that rebuilding an index reproduces it. You cannot diff two builds, you cannot cleanly attribute a retrieval regression to a corpus change versus a chunker change, and "we rebuilt the index" becomes a change with unknown blast radius. **Re-embedding coupling.** Changing the chunker changes every chunk, which changes every vector, which means a full re-embed of the corpus and a full re-run of the evaluation. The chunker prompt and the model version become versioned artefacts of the index, and must be recorded alongside it. **Review burden.** For high-stakes corpora — clinical guidance, regulated procedures, safety instructions — machine-proposed boundaries that split a warning from the action it qualifies are a real hazard. Review makes the technique safe and makes it slow, which is only tolerable at small scale. ## The conditions under which it pays Three things need to be true at once. **Structure is genuinely absent.** If documents are Markdown with consistent heading levels, the structure is already the answer and a splitter gets it for free. Agentic chunking earns its keep on material that has lost its structure: PDF-converted onboarding guides with inconsistent headings, meeting transcripts, scanned procedures, wiki pages written by fifty people with no convention. **Scale and churn are modest.** Hundreds to low thousands of documents, updated on a cadence of weeks or months, is the sweet spot. The per-document cost is bounded and review is feasible. **You have diagnosed a boundary problem.** This is the condition teams skip. Before paying for a smarter chunker, sample real retrieval failures and classify them: was the answer passage missing from the index, ranked too low, split across two chunks, or retrieved and then misread? Only the third category is a boundary problem. If most failures are ranking failures, a better retrieval or ranking stage pays more per unit of effort than any chunker will. ## The cheaper ladder to try first Increase overlap so a fact straddling a boundary appears in both chunks. Split on whatever structure does exist, even partially. Attach document-level metadata to each chunk so filters can narrow before ranking. Fix the obvious pathologies — chunks that are one line long, chunks that are five thousand tokens, chunks consisting entirely of a table's header rows. In many corpora these recover most of the achievable gain at a fraction of the cost, and they leave the index deterministic. ## Governance if you do adopt it Version the chunker as part of the index: prompt text, model identifier, parameters, run date. Pin the model rather than tracking a moving alias, because a provider-side update would otherwise re-chunk your corpus without a deploy. Keep the pre-agentic chunking available so you can A/B and roll back. Sample proposed boundaries for review, weighted toward the highest-stakes documents rather than uniformly. And gate the change on the same end-to-end evaluation you would use for any retrieval change — boundary quality that does not move answer accuracy is a cost you took for nothing. ## What a principal-level answer sounds like It is not "agentic chunking is better". It is: here is the failure I measured, here is why the cheap fixes did not close it, here is the corpus size and churn that makes per-document inference affordable, here is what I gave up in reproducibility, here is how the chunker is versioned so a rebuild is a deliberate act, and here is the accuracy delta that justified it. A candidate who reaches for the model-in-the-loop option without that chain has skipped the part of the job that is actually hard.