How does Cohere's v2/rerank handle a document longer than max_tokens_per_doc?

## The parameter and its default Cohere's v2 rerank request accepts `max_tokens_per_doc`, which caps how much of each supplied document the model considers. The default is 4096 tokens, which also reflects the context the rerank model works within — query plus document have to fit. Documents shorter than the cap are scored in full. Documents longer than it are **truncated**: the model scores the retained prefix and never sees the rest. ## What changed from v1 The v1 rerank API had a different knob, `max_chunks_per_doc`. Its behaviour was fundamentally different: an over-long document was split into chunks internally, each chunk was scored against the query, and the document's score was taken from its best chunk, up to the configured chunk ceiling. That meant a long document with one highly relevant paragraph deep inside could still surface. The v2 API removed that mechanism. There is no server-side chunking; there is only truncation. This is the single most consequential difference for anyone porting a v1 pipeline, and it is dangerous precisely because it is not a breaking change at the type level — you drop one parameter, add another with a similar name, everything still returns 200, and recall on long documents quietly falls off. ## Why truncation hurts and how it shows up Long documents put their conclusions in inconvenient places. A support article buries the fix under three paragraphs of symptoms; a contract puts the liability clause on page nine. Under truncation, the query-relevant span is either inside the retained window (fine) or outside it (the document scores as though it were irrelevant). The failure is asymmetric and invisible: no error, no flag, just a good document ranked below a mediocre short one whose entire text was read. Symptoms in production look like: short FAQ entries dominating the top of every ranking; a known-good long document never appearing however the query is phrased; rerank *hurting* measured quality on a corpus of long files while helping on a corpus of short ones. ## The right shape: chunk before you rerank The durable answer is that reranking operates on passages, not whole files. Split at ingestion into passages that are meaningful on their own — a section, a few hundred tokens with overlap, a table plus its caption — and index those passages. Then: 1. Recall returns passages. 2. Rerank scores passages against the query. 3. You keep the top passages, dedupe by parent document if one document contributed several, and pass those passages into the prompt. This alignment matters beyond truncation: the unit you score is the unit you feed the model, so a high relevance score actually predicts that the retained text answers the question. Reranking a whole document and then feeding the model a different slice of it breaks that link. ## Tuning the parameter itself Lowering `max_tokens_per_doc` below the default is a deliberate trade: it caps the text the model reads per candidate, which reduces the work per document, but it increases the chance of cutting the answer off. Raising it is bounded by the model's context. In practice, if you find yourself wanting a big value, that is a signal your passages are too large and the chunking, not the parameter, is what needs work. ## Migration checklist When moving a v1 rerank pipeline to v2: search the codebase for `max_chunks_per_doc` and treat every hit as a design question, not a rename; measure the token length distribution of your candidates against the 4096 default; add explicit passage chunking wherever documents exceed it; keep parent-document ids on each passage so you can dedupe after ranking; and re-run your retrieval evaluation set before and after, because this change moves quality without moving any status code.