Why can an embedding model silently ignore half of a 900-token chunk?

## The mechanism An encoder is compiled around a maximum position count. Positions beyond that have no positional representation, so the model physically cannot consume them. The tokenizer therefore enforces the limit before the model runs, and the near-universal default is **truncate**: keep the first N tokens, drop the rest, proceed. The consequence is stated simply and is worth saying exactly this way in an interview: *the vector represents the prefix, not the document.* Everything after the cut had zero influence on the numbers you stored. ## Why it stays invisible This is a silent-corruption failure, the worst kind in a retrieval pipeline: - The output vector has the normal dimensionality, so no shape check fires. - Similarity scores stay in their usual range, so no threshold looks odd. - No exception, no warning field, no status code — a truncated call and a clean call are indistinguishable in logs. - Retrieval still returns results. They are simply the wrong ones, or missing the right one, for any query whose answer lived in the tail. The user-visible symptom is a system that answers well about the beginnings of documents and inexplicably "doesn't know" things that are demonstrably in the corpus. Teams usually blame the chunking strategy, the reranker or the model's reasoning long before they check token counts. ## Why token counts surprise people Chunkers are usually written against characters or words because those are what a splitter naturally counts. The model's limit is in **tokens**, and the ratio between them is not a constant. Subword tokenizers keep frequent words whole and split rare ones into pieces. For ordinary English prose the ratio is roughly three-quarters of a word per token, so a rule of thumb of ~400 words fitting in 512 tokens feels safe. Then reality intervenes: - **Domain jargon** — pharmaceutical names, legal Latin, medical terms, taxonomies — is rare in the tokenizer's training data and fragments into many pieces. - **Identifiers and codes** — SKUs, order numbers, UUIDs, error codes, file paths — fragment worst of all, sometimes one token per two or three characters. - **Source code** — indentation, punctuation and camelCase identifiers each consume tokens. - **Non-Latin scripts** — a tokenizer whose vocabulary was built mostly on English may spend several tokens per character on Japanese, Korean or Devanagari. - **Tables and markup** — pipes, tags and repeated delimiters inflate counts dramatically. So a chunk that a word-count check calls "400 words, fine" can be 700+ tokens in a catalogue of part numbers, and half of it disappears. Also remember the model may add special tokens of its own, so the budget for your content is slightly under the nominal maximum. ## Detecting it The check is cheap and belongs at ingest, before anything is indexed: 1. Tokenize each chunk with **the same tokenizer the embedding model uses** — not an approximation, not a different model's tokenizer. 2. Compare the length against the model's documented maximum sequence length. 3. Record the distribution of chunk token lengths as a pipeline metric, and alert on the tail. A histogram piling up exactly at the limit is the signature of mass truncation. 4. Re-run this whenever you change chunker, corpus or embedding model — a new model can change both the limit and the tokenizer. A useful sanity test: embed a document, then embed only its second half, and check whether the two vectors are suspiciously unrelated while the first half's vector is nearly identical to the whole document's. That pattern is truncation caught red-handed. ## Fixing it - **Chunk to the model's real limit, measured in tokens**, with headroom for special tokens. - **Use a long-context encoder** if your unit of meaning genuinely does not fit. Encoders accepting 8k tokens and beyond are widely available as of mid-2026. Note the trade-off: a longer window is not free quality — pooling a very long passage into one vector averages away specifics, so a single vector for an entire long document is often *worse* at retrieval than several vectors for its parts, even when the model can technically ingest it. - **Split and aggregate** when a document must stay one logical unit: embed sub-chunks, store them individually, and let retrieval work at that granularity. - **Fail loudly instead of truncating** in your own ingest code — treat an over-length chunk as a pipeline error rather than accepting the tokenizer's silent default. ## The general lesson An embedding pipeline has several steps that degrade quality without producing an error, and truncation is the sharpest of them. Any stage that can silently discard input deserves an explicit assertion at ingest, because the cost of finding it later is measured in a full re-index of the corpus.