In DeepEval, is a high HallucinationMetric score good or bad?

## Two families, one argument name DeepEval's built-ins split into two groups that behave in opposite directions but share an identical-looking constructor. **Higher is better (quality metrics).** `AnswerRelevancyMetric`, `FaithfulnessMetric`, `ContextualPrecisionMetric`, `ContextualRecallMetric`, `ContextualRelevancyMetric`. The score is a proportion of goodness — how much of the answer was relevant, how many claims were supported, how much of the retrieved context was on topic. 1.0 is perfect, and the case succeeds when the score is at or above the threshold. **Lower is better (safety / error metrics).** `HallucinationMetric`, `ToxicityMetric`, `BiasMetric`. The score is a proportion of badness — how much of the output contradicts the supplied context, how much of it is toxic or biased. 0.0 is perfect, and the case succeeds when the score is at or below the threshold. Both families expose `threshold`, both default it to 0.5, and both report through `is_successful()`. Nothing in the API surface tells you which direction you are in; you have to know the metric. ## The mistake this produces The failure is not exotic, it is the natural one. Someone has internalised "higher threshold means stricter" from the relevancy metrics and applies it uniformly. Writing `AnswerRelevancyMetric(threshold=0.9)` genuinely is strict — it demands 90% relevance. Writing `ToxicityMetric(threshold=0.9)` is the opposite: it tolerates a toxicity score up to 0.9 before failing anything. The second line reads like tightening and is in fact almost complete permissiveness. Worse, it produces no visible symptom: the suite goes green, and it goes green precisely on the cases you built the metric to catch. A safety metric that never fires looks exactly like a system that is behaving. The mirror-image mistake is setting a hallucination threshold very low as a *loosening* gesture and then wondering why every case fails. ## How to reason about the number For the lower-is-better metrics, read the threshold as a **tolerance**, not a bar. `ToxicityMetric(threshold=0.0)` means "any detected toxicity fails this case", which for a user-facing product is often the right setting and is a perfectly defensible answer in an interview. `HallucinationMetric(threshold=0.2)` means "up to a fifth of the supplied contexts may be contradicted before I call this a failure" — sometimes reasonable on noisy multi-document cases, rarely reasonable on a single authoritative context. For the higher-is-better metrics, read the threshold as a **bar**, and set it below the score you actually observe rather than at it, because the judge is an LLM and its output moves between runs. A threshold pinned to a single observed score turns normal judge variance into random CI failures. ## Reading the score itself Because the score is a proportion in both families, it carries information beyond pass/fail. A hallucination score of 0.5 on a case with two contexts means one of them was contradicted — that is a much more actionable statement than "failed". Turning on the reason string makes that concrete, giving you the judge's account of which claim conflicted with which context. When you are triaging a failing safety case, the reason plus the raw proportion usually locates the problem faster than re-running with a different threshold. ## What to say in the interview Three beats: name the two families, state that threshold semantics invert between them, and give the concrete consequence — that a threshold copied from a quality metric to a safety metric silently disables it. If you can add that safety thresholds are usually set at or very near 0.0 while quality thresholds are set with margin below observed scores, you have covered both the mechanic and the judgment.