Why can a RAG answer score perfect faithfulness and still be wrong?

## The metric's reference point Every evaluation metric answers "consistent with what?" For faithfulness the answer is: the retrieved context passed to the generator. That choice is deliberate and useful — it isolates the generator's behaviour from the quality of the corpus — but it means the metric inherits whatever the corpus says. Truth is never consulted. A system whose corpus is wrong will have a perfectly grounded, perfectly wrong product, and the faithfulness dashboard will be green throughout. ## Three ways a fully faithful answer is wrong ### The source is wrong or out of date The most common case in enterprise RAG. Corpora accumulate superseded versions, drafts, region-specific variants, deprecated runbooks, and content written by users rather than by the organization. Retrieval finds the passage that best matches the query, not the one that is currently authoritative — an old version often matches *better*, because deprecated documents tend to state things plainly while the current one is hedged. The generator paraphrases it accurately. Every claim is entailed. The score is 1.0 and the user acts on a rule that changed last quarter. ### The sources conflict When the context contains two passages that disagree, any answer the model produces is entailed by *some* retrieved passage, so a naive per-claim entailment check passes whichever branch it takes. Worse, the choice is often arbitrary — driven by position in the context or by phrasing similarity rather than by recency or authority. The metric cannot distinguish "resolved the conflict correctly" from "picked one at random", because both are grounded. Detecting this needs a separate signal: flag contexts whose passages contradict each other, and treat unacknowledged conflict as a defect regardless of the faithfulness score. ### The answer is grounded but useless There is a degenerate corner where the metric can be optimized without improving anything. An answer that makes no factual claims — a refusal, a restatement of the question, a pure hedge — has nothing to be unfaithful about and scores 1.0 or undefined. Tighten the grounding instruction hard enough and faithfulness climbs while the product gets worse, because the model now abstains on questions it used to answer well. This is why abstention rate must be reported next to faithfulness: a metric that improves while the abstention rate doubles has not improved. A related corner is omission. An answer can be complete-looking, every claim supported, and materially misleading because it never mentions the exclusion, the exception or the eligibility condition sitting in the next paragraph. Faithfulness scores what was said; it has no opinion about what was left out. ## What to measure instead, alongside The three axes are genuinely separate and none substitutes for another: - **Faithfulness** — answer versus retrieved context. Catches generator hallucination. - **Answer relevance** — answer versus question. Catches drift and non-responsiveness. - **Answer correctness** — answer versus a gold reference answer. Catches everything the first two are blind to, including bad sources. Only the third requires labels, which is why teams skip it and then discover the blind spot in production. You do not need labels at scale: a modest curated set of questions with human-verified answers, refreshed as the corpus changes, is enough to catch a systematic corpus problem. Correctness low while faithfulness is high is the signature of a source problem, and it is unmistakable once you plot the two together. ## Fixes that live outside the generator Because the cause is upstream, so are the remedies: - **Freshness and versioning.** Carry effective dates and version metadata on chunks, filter or down-rank superseded content at query time, and delete rather than archive-in-place where you can. - **Authority ranking.** Not all sources are equal. Official policy outranks a support-ticket transcript; make that explicit in ranking or filtering rather than hoping the embedding sorts it out. - **Deduplication.** Near-identical chunks from multiple versions are the primary generator of silent conflicts. - **Conflict surfacing.** When retrieved passages disagree, prefer an answer that names the conflict over one that silently picks a side — and measure how often that happens. - **Ingestion hygiene as an ongoing job.** Corpus rot is continuous; a one-time cleanup regresses within months. ## How to talk about this in an interview The strong answer states the reference point in one sentence — faithfulness is measured against the retrieved context, not the world — then names the concrete cases, then says what you would add to catch them. It also concedes the metric's virtue rather than dismissing it: separating generator behaviour from corpus quality is exactly what makes faithfulness a good *component* metric. The mistake is treating a component metric as a product metric.