Why can near-duplicate documents in a prompt hurt accuracy even when the right one is included?

## Two different long-context failures It is worth separating the two ways a long prompt goes wrong, because they need different fixes. The first is **dilution**: the relevant span is present but the model does not locate or use it, which is the classic long-context and position-bias story. The second is **competition**: the model locates relevant-looking material perfectly well, but several passages look relevant and it uses the wrong one. The second failure is more dangerous in production because it does not look like a failure. There is no hedging and no missing answer; there is a fluent, well-grounded-sounding response quoting the wrong version of a document. ## Why near-duplicates are the worst case Retrieval systems and attention both work on similarity. A near-duplicate is, by construction, maximally similar to the correct document — that is what makes it a near-duplicate. So it scores highly at retrieval time and it competes strongly at attention time. Meanwhile the feature that would distinguish it, its *status* (current, draft, superseded, site-specific), is usually not expressed in the text of the passage at all. It lives in a filename, a header page, a database column, or nowhere. Consider a study assistant answering questions about participant consent. Nine versions of the consent form are pasted in: an original, five site-specific variants, two superseded revisions, and the current master. A question about whether re-consent is required after a protocol amendment matches all nine roughly equally. The model answers from whichever version it weighted most — plausibly the one nearest an edge of the prompt, or the one whose phrasing best matches the question — and there is no signal in the text saying that seven of the nine are void. Adding the correct document did not help; adding the other eight actively hurt. This is why "just raise k, recall is what matters" is a mistake. Increasing the number of retrieved passages raises the chance the right one is present and simultaneously raises the number of confusable wrong ones. Past some point the second effect dominates and end-to-end accuracy falls while retrieval recall keeps rising — a metric divergence that surprises teams optimizing the retriever alone. ## How to control it **Deduplicate before assembling.** The cheapest fix is to never place nine versions in the prompt. Near-duplicate detection at index time — hashing, shingling, or an embedding-similarity threshold — collapses families of variants into one representative, with the others reachable only if explicitly requested. **Make status a first-class field, not prose.** If versions genuinely must coexist, filter on structured metadata (effective date range, status, site) before the model ever sees them. This turns a reasoning problem the model is bad at into a query predicate that a database is good at. **When you must show conflicts, label them and state the rule.** If the task really is reconciliation, attach a short explicit header to each passage — identifier, version, effective date, superseded-by — and put the precedence rule in the instruction: prefer the latest effective version, and say when versions conflict. Models handle explicit precedence much better than inferred precedence, but only if the metadata is actually in the prompt. **Tune k against the end-to-end answer, not retrieval recall.** The right number of passages is the one that maximises correct final answers, and it is often smaller than the retriever's optimum. Measure both curves and look for the crossover. ## Measuring it The experiment that isolates this effect holds length constant and varies confusability. Fix the total prompt at, say, 60K tokens. In arm A, the correct consent form sits among unrelated filler. In arm B, the same correct form sits among eight near-duplicate versions padded to the same total length. Same length, same question, same position — the only difference is competition. The accuracy gap between the arms is the distractor penalty, cleanly separated from any length effect. Run the same design with graded similarity: unrelated filler, topically-related-but-different documents, then true near-duplicates. The penalty typically grows with similarity, which is the result that convinces people the problem is confusability rather than volume. It also tells you where to spend: if the topically-related arm is fine and only the near-duplicate arm collapses, deduplication is the whole fix and you do not need to rebuild retrieval. ## What not to conclude Do not conclude that more context is always bad. Adding genuinely complementary evidence usually helps. The specific hazard is redundant, confusable material that carries no distinguishing signal — and the specific defence is removing it or labelling it, not shrinking the prompt indiscriminately.