Why split Markdown docs on their heading hierarchy instead of a fixed character count?

## What structure-aware splitting means Structure-aware chunking uses boundaries the document's author already created — headings, sections, list blocks, table rows, fenced code — instead of counting characters or tokens. In Markdown that means splitting on `#`, `##`, `###` levels; in HTML on `<h1>`–`<h3>`, `<section>` and `<article>` elements; in a wiki or docs-site export on whatever section markup it emits. The splitter parses the markup, walks its tree, and emits one chunk per section together with the chain of headings above that section. It is deterministic and cheap: no model calls, no embeddings, reproducible across runs. That matters because you re-run chunking every time the corpus changes. ## Why length-based cuts hurt retrieval A fixed 800-character window is blind to topic boundaries, and three failure shapes follow. First, straddling. One chunk holds the last two paragraphs of "Rate limits" and the first paragraph of "Error codes". A single embedding vector has to represent both, so it lands somewhere between the two subjects and ranks poorly for either question. Second, splitting a unit that only makes sense whole. A definition ends up in chunk 7 and the example that illustrates it in chunk 8. Retrieval returns one of them and the model answers half the question, confidently. Third, orphaned references. The heading is the sentence that says what the text is about. Cut it away and the chunk reads "this endpoint returns a 429 with a `Retry-After` header" — full of pronouns with no antecedent, and containing none of the words a user would actually search for. ## The heading path is the real prize The boundary is useful; the path is more useful. A header splitter knows, for every chunk, the full chain of ancestors: `Billing API > Rate limits > Retry behaviour`. Do two things with it. Prepend it into the chunk text, so it is part of what gets embedded and part of what a keyword index matches. A chunk whose body never repeats the word "billing" becomes findable by a query about billing. Also store it as a structured field alongside the vector. That lets you filter ("only sections under the v2 API"), route, and — most importantly — show a real citation with a deep link back to the source section rather than an anonymous fragment. ## Where it needs help Headings are irregular, so a header splitter alone will not produce chunks that fit your budget. Two policies close the gap. Oversized sections: when one section exceeds the budget, fall back to a length-based split inside that section only, and repeat the heading path on every resulting piece so none of them becomes an orphan. Undersized sections: a page of one-line `###` entries produces dozens of near-empty chunks whose embeddings are noisy. Merge adjacent siblings under the same parent until they reach a useful size, keeping the shared parent path. And some documents have no usable structure at all — OCR'd scans, meeting transcripts, minified HTML, plain text dumps. There the honest answer is either to recover structure first (layout detection, speaker turns, a heading-inference pass) or to fall back to length-based splitting and accept the cost. ## HTML has an extra trap HTML carries structure and noise in the same tree. Navigation menus, cookie banners, footers and sidebars will be dutifully chunked alongside the content if you split naively, and they are near-identical across every page in the corpus — which means near-duplicate embeddings that crowd out real answers. Strip boilerplate before splitting, and keep tables and code elements intact as units rather than letting a text extractor flatten them into unreadable runs. ## How you would know it is working Measure retrieval, not vibes. Build a small set of real questions with the section that actually answers each one, and track whether that section appears in the top-k. Then eyeball the boundaries: print thirty random chunks and ask whether each one reads as a coherent, self-contained passage about a single subject, and whether you can tell what document and section it came from without being told. If the answer to the second question is no, your prefix is missing, not your splitter. ## The mental model Length-based splitting asks "how much text fits?" Structure-aware splitting asks "what is one idea?" The first is about the embedding model's window; the second is about the document. You need both, in that order: cut on structure, then enforce length inside whatever structure gives you.