Why does RAG retrieval use a bi-encoder rather than scoring every document with a cross-encoder?

## Two ways to compare a query and a document Any neural relevance model has to bring a query and a document together somewhere. There are two arrangements, and the choice determines the entire shape of the retrieval system. In a **bi-encoder** (also called a dual encoder) the query and the document are pushed through the encoder *separately*. Each becomes a fixed-length vector — a few hundred to a couple of thousand floats. The relevance score is then a cheap arithmetic operation between the two vectors, almost always a dot product or cosine similarity. In a **cross-encoder** the query and the document are joined into a single input and pushed through the model *together*. The model reads them jointly and emits one relevance score. There is no separable document representation to store: the score exists only for that specific pair. ## Why independence is the whole point Because a bi-encoder encodes a document without knowing the query, the document's vector can be computed at **index time**, long before any user shows up. Ingestion becomes a batch job: chunk the corpus, embed once, write the vectors into a vector index. At query time you pay for exactly one forward pass (the query) plus a similarity search. That search is not a linear scan either. Because every item is a point in the same vector space, an approximate-nearest-neighbour index can prune most of the corpus and return the top *k* in roughly logarithmic or sublinear time. Precomputation plus ANN is what makes semantic search over millions of chunks answerable in tens of milliseconds. ## The cost arithmetic Make it concrete with a four-million-chunk corpus of 800-token chunks. - **Bi-encoder:** four million forward passes *total*, paid once at ingestion (and again only when you change models). Per query: one short forward pass and an index lookup. - **Cross-encoder over the full corpus:** four million forward passes over ~800-token pairs *per query*. Even at a thousand pairs per second, a single query takes over an hour of GPU time. The asymmetry is not a small constant factor; it is the difference between a feasible system and an impossible one. ## What the bi-encoder gives up The independence that buys the speed is also the weakness. A document's vector must be a single summary of everything the chunk might ever be asked about, committed before the question exists. Fine-grained interactions — an exact identifier, a negation, a rare term that decides relevance — get averaged into one point. A cross-encoder, by contrast, lets every query token attend to every document token, and its accuracy advantage on hard pairs is consistently measurable. The standard resolution is a **cascade**: use the bi-encoder for high-recall, low-precision retrieval of maybe the top 50-200 candidates, then spend a query-aware scorer only on that small set. You get corpus-scale reach at bi-encoder cost and pair-level precision where it matters. ## Consequences you inherit by choosing a bi-encoder 1. **The index is tied to a model version.** Vectors from model A are meaningless in a space built by model B. Switching embedding models means re-embedding the whole corpus, which is why the choice is expensive to revisit. 2. **Query encoding sits on the hot path.** Its latency and per-call cost are paid on every request, so a big encoder is much more affordable at index time than at query time. 3. **Both sides must use the same model and the same conventions** — same version, same normalization, same query/document prefix scheme if the model expects one. Mixing them silently degrades ranking rather than erroring. 4. **ANN adds its own approximation.** Even a perfect bi-encoder loses a little recall to the index's search parameters, so retrieval quality is a property of model *and* index together. ## What interviewers listen for The weak answer is "bi-encoders are faster" with no account of *why*. The strong answer names independence-and-precomputation as the mechanism, states the cost arithmetic, and then concedes the accuracy gap honestly instead of pretending the bi-encoder is simply better — because the concession is what motivates the two-stage pipeline every real RAG system ends up with.