Which Ragas metrics run on production traces with no reference answers?

## Why the split matters at all Production traffic gives you three things for free: the question the user asked, the chunks your retriever returned, and the answer your model produced. It never gives you the correct answer, because if you had that you would have shipped it. Ragas's metric catalogue is therefore partitioned by a hard data requirement, and the first thing a senior engineer does with the catalogue is sort it. ## The reference-free set **Faithfulness** takes `user_input`, `response` and `retrieved_contexts`. It asks whether the answer is supported by the chunks that were actually retrieved — a purely internal consistency question, so no ground truth is needed. **ResponseRelevancy** takes `user_input` and `response` plus an embeddings object. It has the judge reverse-generate questions from the answer and measures how close they are to the real question. Again, self-contained. **LLMContextPrecisionWithoutReference** takes `user_input`, `response` and `retrieved_contexts`. This is the deliberate substitution: where the `WithReference` class asks "was this chunk useful for arriving at the ground-truth answer?", the `WithoutReference` class asks "was this chunk useful for arriving at the answer we actually produced?". Same ranking-aware precision computation, different judging target. **AspectCritic** and **RubricsScore** are reference-free or not depending on you. An AspectCritic whose `definition` reads "Does the response contain any claim not present in the retrieved context?" runs on live traffic. One that reads "Does the response match the ground truth?" does not, because it names a field that is empty. ## The reference-requiring set `LLMContextPrecisionWithReference`, `LLMContextRecall`, `FactualCorrectness` and `NoiseSensitivity` all read `reference`. These are your offline metrics. They answer questions the reference-free set structurally cannot — most importantly recall-shaped questions, since "did we retrieve everything needed" is unanswerable without knowing what was needed. ## The two-suite pattern The arrangement that survives contact with production is two distinct suites with different cadences. One suite runs the reference-free metrics over a sample of live traces. It is continuous, cheap per sample because you control the sample rate, and it is a monitor: it tells you *something changed*, not *the change is bad*. This suite catches retrieval drift, prompt regressions and a model swap that broke grounding. The other suite runs the full catalogue over a curated labelled dataset — a few hundred examples where a human wrote the `reference`. It is a gate: you run it before a release, compare against the previous run, and it is the only place recall-shaped and correctness-shaped numbers exist. Its size is bounded by annotation budget, not by compute. ## The trap, stated plainly The move that looks clever and is not: take the production `response`, write it into `reference`, and unlock the reference-requiring metrics on live traffic. Every one of those metrics then compares the answer to itself. `FactualCorrectness` returns approximately a perfect score forever. `LLMContextRecall` reports that retrieval covered everything the answer needed — which is trivially true, because the answer was written from what was retrieved. You have manufactured a dashboard that cannot go red, which is strictly worse than having no dashboard, because someone will trust it. A weaker variant of the same trap is generating `reference` with a stronger model and treating it as ground truth without ever auditing it. That can be legitimate — but only if you have sampled and hand-checked the generated references, and only if you are honest in the reporting that the number is model-versus-model agreement, not correctness. ## Practical selection Given a leaf-level metric choice, the decision procedure is short. Do I have `reference` for this data? If yes, the full catalogue is open. If no, take Faithfulness for grounding, LLMContextPrecisionWithoutReference for retrieval usefulness, ResponseRelevancy for on-topic-ness, and an AspectCritic or two for the domain-specific properties you care about — refusal behaviour, tone, presence of a citation. Then accept that no reference-free metric can tell you the answer was *right*, only that it was consistent with what you retrieved, and route the correctness question to the labelled suite.