How does claim-level entailment scoring measure a RAG answer's faithfulness?

## The idea Asking a judge "is this answer grounded?" produces a verdict you cannot act on. Claim-level entailment scoring turns faithfulness into many small, checkable questions instead of one big vague one. It borrows directly from fact-verification research: decompose the text into atomic claims, verify each against a source, aggregate. ## Stage 1 — decomposition A model rewrites the answer as a list of atomic claims. "Atomic" means one assertion per claim; "self-contained" means pronouns and references are resolved, because each claim will later be shown to a verifier without the rest of the answer. "Your policy covers it, but you'd pay the first $500" becomes two claims: *the policy covers water damage* and *the policyholder pays a $500 deductible*. Decomposition is where most of the error budget goes. Over-splitting produces fragments that are not independently verifiable and dilutes the denominator — twelve trivial claims from a three-sentence answer means one hallucination costs you 1/12 rather than 1/3. Under-splitting hides an invented detail inside a mostly-true compound sentence. Non-factual material — hedges, restatements of the question, offers to help further — should be dropped rather than scored, or every polite closing sentence becomes an unsupported claim. ## Stage 2 — verification Each claim is checked against the retrieved context, with three useful verdicts rather than two: - **Supported / entailed** — the context implies the claim. - **Contradicted** — the context says something incompatible. This is the severe case; the model overrode its own evidence. - **Not addressed** — the context is silent. The claim may be true in the world, but it came from pretraining, not from retrieval. Collapsing the last two into one "unsupported" bucket loses real information: contradiction usually points at a prompt or model problem, while a pile of not-addressed claims usually points at thin retrieval. Two implementations dominate. A dedicated NLI model is cheap, fast and deterministic, but works best on short sentence pairs and struggles when support is spread across several passages. A judge model with an explicit rubric handles multi-hop and paraphrased support far better, costs a call per claim, and needs its own validation before you trust it. Whatever the verifier, it must be given only the retrieved context — not the wider corpus, not the internet — otherwise it certifies claims the generator never had grounds for. ## Stage 3 — aggregation The standard score is supported claims over total claims. Simple, comparable across runs, and lossy in specific ways worth naming: - **Severity is flattened.** An invented deductible amount and an over-general adjective both cost one claim, though only one will generate a complaint. - **The denominator is gameable.** Verbose answers that pad with easily-supported restatements score higher than terse ones with the same single hallucination. - **Refusals score perfectly.** An answer with zero claims is either 1.0 or undefined; a system that abstains more looks more faithful. Common mitigations: report the *count* of unsupported claims alongside the fraction, weight contradictions more heavily than not-addressed, and gate on "zero contradicted claims" rather than on an average, since one confidently wrong number in a regulated domain is not offset by nine correct ones. ## What claim-level scoring still misses Even a perfect per-claim pass leaves gaps. **Omission** is invisible: an answer stating the policy covers water damage, every claim entailed, can be dangerously incomplete because it never mentions the exclusion in the next paragraph — nothing false was said. Relatedly, individually-supported claims can be **arranged** into a misleading whole through selection, ordering or emphasis; entailment is checked claim by claim, so the composition is never examined. **Implicit claims** — presuppositions carried by phrasing rather than asserted outright — often escape the decomposer. And faithfulness says nothing about whether the context itself was right, so a fully-supported answer built on a superseded document scores 1.0. ## When whole-answer scoring is the better trade Holistic scoring — one judge call, one verdict, optionally a short justification — costs a fraction as much and is adequate for coarse monitoring of high traffic, for smoke checks in CI, and as a cheap first tier that escalates only suspicious answers to full decomposition. Claim-level scoring earns its cost where you need to *localize* a failure: regression triage, comparing two prompts, or any domain where you must be able to point at the exact sentence that was not supported. Many teams run holistic scoring online and claim-level scoring offline on a sample, which keeps the diagnostic power without paying for it on every request.