In RAG, when relevance alone picks the wrong passages, how do you design selection policy?

## Relevance is a proxy, not the objective Every scoring stage in a retrieval pipeline optimises textual and semantic similarity to the query. That is a proxy for the real objective, which is: *which passages, read together, let the model produce a correct, current, appropriately sourced answer?* Wherever those diverge, relevance ranking confidently picks the wrong thing, and no amount of reranker quality fixes it — because the reranker is doing its job correctly. The divergences recur across domains: - **Staleness.** A superseded policy or an expired contract often matches a question *better* than its replacement, because the replacement is written in newer language. Similarity has no notion of "current". - **Authority.** A canonical runbook and a three-year-old chat thread can be near-identical in wording; only one should ground an answer. - **Coverage.** A question with several facets — cause, remedy, and who to notify — can be filled entirely by passages about the cause, because those are the most on-topic. - **Entitlement.** The best passage may be one this user is not permitted to see, which is a correctness and compliance issue rather than a ranking one. - **Confidence.** Sometimes the right selection is the empty set. ## Make selection its own stage The design move is to stop treating "take the top m" as a line of code and start treating selection as a stage with its own inputs, rules and tests. It consumes ranked, scored candidates plus their metadata, and emits the final set plus a record of why each passage was kept or dropped. Typical rules in such a layer: - **A relevance floor**, so a slot is never filled just because it exists. - **Hard filters**, applied before anything else: access control, tenancy, jurisdiction, document status. These are correctness constraints, not preferences, and pushing them into the retrieval query is usually better than filtering afterwards. - **Recency policy**: a hard cutoff where old material is dangerous, or a soft decay that discounts age when it is merely less useful. - **Authority tiers**: a preference ordering over sources, or a tie-break that prefers the canonical copy when two candidates say the same thing. - **Caps**: at most so many passages per document or per source, which prevents one long document from occupying the whole set. - **Reserved slots**: for multi-facet questions, require the set to include at least one passage of each required kind. ## The costs, stated honestly A policy layer is not free and a principal-level answer says so. **Ranking metrics will get worse.** You are deliberately demoting the highest-scoring candidate. If the team's dashboard is a ranking metric, the policy will look like a regression forever. Move the headline metric to end-to-end answer quality before shipping the policy, or it will be reverted by someone reading the dashboard. **Rules accumulate and interact.** Each incident tempts a new rule. Six months later, no one can predict what the selector will do, and the rules contradict each other. Mitigate with a fixed evaluation order, a bounded rule count, an owner, and a periodic review that deletes rules whose motivating incident is gone. **Rules encode assumptions that expire.** "Prefer documents from the compliance space" is right until the team reorganises. Version the policy alongside the corpus, and treat it as a configuration artifact with a changelog rather than as prompt text. **Hard-coding beats the model's judgment — sometimes wrongly.** A strict recency cutoff makes historical questions unanswerable. Where the right behaviour depends on the query ("what did the policy say last year?"), the policy must branch on query intent rather than apply a blanket rule, which means intent classification becomes a dependency with its own error rate. ## Who owns it Selection policy is where business rules meet the pipeline, so it should be legible to people who are not on the retrieval team. Compliance cares that superseded documents cannot ground an answer; support leadership cares that the canonical runbook wins; legal cares about jurisdiction filters. Expressing the policy declaratively — a small, ordered, readable set of constraints with a version — lets those stakeholders review it, and lets you answer the audit question "why did the system cite that document?" from a log rather than from a re-run. ## Evaluating it Evaluate outcomes, not order. - **Slice the answer eval by the cases the policy targets**: questions with a superseded predecessor in the corpus, multi-facet questions, questions with a permission-restricted best match. Aggregate scores will hide the exact behaviour you built the policy for. - **Guard against collateral damage** with a broad regression set, since a rule aimed at one failure mode routinely degrades unrelated queries. - **Log the drop reason per candidate.** When an answer is wrong, you need to know whether the passage was never retrieved, was ranked low, or was retrieved, ranked well and then rejected by policy — three different fixes. - **Feed incidents back.** Each wrong-source incident should end with either a new rule and its eval case, or an explicit decision not to add one. The strongest version of this answer accepts that the policy is a living business artifact: it will grow, it must be pruned, its ranking metrics will look bad, and its justification lives in end-to-end outcomes and incident review.