What does a cross-encoder reranker buy over the vector index's own ranking?

## Bi-encoder versus cross-encoder A **bi-encoder** encodes each text independently into a vector. Chunks are embedded once at ingestion and stored; at query time only the query is embedded, and the search is a nearest-neighbour lookup in an index. Because the two encodings never see each other, the chunk's representation must be a single fixed summary that works for every possible future query. That is a strong compression, and it is where the coarseness comes from: two chunks that both sit near the query in the space may differ enormously in whether they actually answer it. A **cross-encoder** takes the query and one candidate concatenated as a single input and produces a relevance score, with attention running across both texts. It can see that the query's constrained noun phrase appears as the subject of the candidate's key sentence, that a negation flips the meaning, that a date qualifier does not match. Nothing is precomputable — every query-candidate pair is a fresh forward pass — so the cost is N model calls for N candidates. That asymmetry dictates the architecture. You cannot cross-encode millions of chunks per query, so you use the cheap index to cut the corpus to a hundred candidates and spend the expensive model only there. ## Where it sits in the pipeline In a hybrid setup the shape is: lexical retriever and dense retriever each return their top candidates, the two lists are fused, the fused head is reranked, and the best few are handed to the generation step. In an e-discovery search the concrete numbers might be 100 fused candidates reranked down to 12 shown to the paralegal or fed to the model. ## Precision, not recall This is the point interviewers actually probe. Reranking reorders a set; it cannot conjure a document into it. If the responsive chunk ranked 340th in the first stage, retrieving 100 candidates means the reranker never sees it and the pipeline fails no matter how good the reranker is. So the two stages have different metrics: measure **recall@N** for the retrieval stage — what fraction of known-relevant chunks appear anywhere in the N candidates — and measure ordering quality, such as nDCG@10, for the reranker. Optimizing the reranker while first-stage recall@100 sits at 70% is wasted effort; the ceiling is 70%. ## Cost and latency Reranking adds a synchronous stage to every query. A small cross-encoder over 100 short passages, batched on a GPU, typically lands in the tens of milliseconds; a larger model, longer chunks, a CPU deployment or a hosted rerank service pushes it into the hundreds. Cost scales with candidates times chunk length, so the two dials are N and how much text each candidate contributes. Practical levers: rerank fewer candidates for interactive paths and more for batch or high-stakes ones; truncate each candidate to the region that matched; batch aggressively; and cache reranked results for repeated queries. ## Choosing N N is an explicit recall-versus-latency trade. Derive it, do not guess it: on a labelled query set, plot first-stage recall@N as N grows. The curve rises steeply then flattens. Pick N just past the knee, because beyond it you are paying linear rerank cost for negligible additional recall. Re-derive N whenever the embedding model, the chunking or the corpus changes, since the shape of that curve is a property of the retrieval stage. ## Alternatives and variants **Late-interaction models** (the ColBERT family) sit between the two extremes: they store per-token vectors and compute a cheap interaction at query time, buying much of the cross-encoder's discrimination at lower query cost, at the price of a substantially larger index. **LLM rerankers** prompt a general model to score or order candidates, sometimes listwise over a whole batch. They can be very good and need no specialized model, but they are slower and dearer than a purpose-built cross-encoder, and they inherit judge-style biases such as sensitivity to candidate order. **Feature-based reranking** — blending the relevance score with recency, authority, custodian or access-tier signals — is often what production actually needs. A pure semantic reranker has no idea that a 2019 draft matters less than a 2024 signed version. ## Failure modes to watch The reranker is another model with its own domain sensitivity: one trained on web passages may underperform on dense legal or clinical prose, and it is worth evaluating on your own labelled set rather than trusting a leaderboard. It also introduces a second point of drift — when retrieval quality changes, the reranker's input distribution changes with it. And because it sits synchronously in the request path, its failure mode must be decided in advance: on timeout, degrade to the fused first-stage order rather than failing the query.