Why can an MS MARCO-trained cross-encoder underperform your bi-encoder on a specialist corpus?

## Relevance is learned, not universal A cross-encoder does not compute relevance from first principles. It was trained on a dataset of queries, passages and relevance judgements, and what it learned is *that dataset's* notion of which passage answers which query. The dominant public training set for passage rerankers is MS MARCO, built from real Bing queries with human-judged passages. Everything about it shapes the resulting model: queries are short natural-language questions, passages are general web prose of a few sentences, and relevance means roughly "this paragraph contains the answer a general reader wanted". That is an excellent prior for a general question-answering assistant over web-like content. It is a much weaker prior for a corpus that looks nothing like the web. ## What goes wrong out of domain Take a veterinary formulary. Passages are dense, semi-structured entries: drug name, species, dosage per kilogram, route, contraindications, often as tables flattened into text. Queries are clinical shorthand — abbreviations, species plus drug plus a numeric question — not fluent web questions. Several failure paths open at once: - **Vocabulary drift.** Terms that carry the entire meaning (a drug abbreviation, a species code) are rare or absent in the training distribution, so the model has weak representations for exactly the tokens that decide relevance. - **Passage shape.** The model learned to reward flowing prose that states an answer. A dosage table that *is* the answer may look, to it, like a low-quality passage. - **Query shape.** Terse keyword-ish queries are unlike the natural-language questions the model saw, so its query encoding is less reliable. - **A different relevance criterion.** In many specialist settings, relevance means "applies to this exact species/jurisdiction/version", a constraint the general model never learned to enforce and will happily ignore. Meanwhile the bi-encoder in your first stage may have been adapted to the domain, or may simply be doing lexical-adjacent matching that happens to work. So you get the counterintuitive result: the strictly more powerful architecture makes the results worse. ## The damage is asymmetric A reranker only ever *reorders*, so a bad reranker cannot improve recall but can absolutely destroy precision at the top. If it demotes the single correct passage from rank 1 to rank 8 and you pass three passages to the generator, the answer is now unsupported — a worse outcome than not reranking at all. This is why "we added a reranker and quality went down" is a real and common report, not a misconfiguration story. ## How to find out before your users do Make unreranked order a permanent baseline in the eval, not a one-off check. Score a labelled query set both ways at the depth you actually pass to the generator. Two things to insist on: 1. **Per-query deltas, not just the mean.** A +2 point average can consist of many small wins and a handful of catastrophic losses, and the losses are what users notice. Count queries where the correct passage's rank got worse. 2. **Slices.** Break the eval by query type, by document family, by whether the query contains domain abbreviations. Out-of-domain damage is rarely uniform; it concentrates in the slices furthest from web prose. Also watch the score distribution. Reranker scores that all cluster tightly, or that rank near-duplicates above the one distinguishing passage, are a signal the model has no real opinion on your text. ## The repair options, roughly in order of cost - **Skip reranking.** If the first stage is already domain-adapted and the reranker does not beat it, the cheapest, fastest and most reliable configuration is the one without it. Say this out loud in an interview; candidates rarely do. - **Try a different checkpoint.** Multilingual and newer general rerankers differ substantially in how they transfer. This is a cheap experiment with a real chance of working. - **Fine-tune on domain pairs.** The high-value move. Collect queries (from logs, from support tickets, or synthesized from documents by an LLM and then human-checked) with a known relevant passage, and mine **hard negatives** from your own retriever's top results — passages that scored well but are wrong. Training against those teaches the distinctions your corpus actually requires. Random negatives teach almost nothing, because they are trivially separable. - **Rebuild the label set as you learn.** Domain relevance criteria surface during labelling: whoever adjudicates will discover that species, dosage form or document version is decisive. Encode that in the labelling guidance and the fine-tuning data. A fine-tuned reranker also carries an ongoing cost: it is now a model you own, version, evaluate and retrain when the corpus shifts. That is a legitimate reason to prefer the no-reranker configuration when the gain is small.