How does the rubric argument change scoring in DeepEval's GEval metric?

## The problem rubrics solve Ask a model to "score this answer from 0 to 1" and it will comply, but the number is anchored to nothing. Its notion of 0.6 is not stable across cases, across prompt phrasings, or across model versions. You see this as scores that cluster oddly, as a suite whose average shifts when you change nothing important, and — worst — as an inability to explain to a colleague what a given score means. `rubric` is the fix inside the tool: instead of leaving the scale implicit, you define the bands and describe each one. ## The shape of it `GEval` accepts `rubric` as a list of `Rubric` objects, imported from `deepeval.metrics.g_eval`. Each carries: - **`score_range`** — an inclusive integer band on a 0-10 scale, for example `(3, 6)`. - **`expected_outcome`** — prose describing the output that belongs in that band. The bands should tile the whole 0-10 range without gaps or overlaps, or the judge has to decide where an in-between output goes, which reintroduces exactly the ambiguity you were removing. The metric's reported `score` remains a value between 0 and 1, as with every DeepEval metric, and it is that normalized value that is compared with `threshold`. So a rubric whose passing band starts at 7 out of 10 pairs naturally with `threshold=0.7`. ## Writing bands that work The descriptions carry the weight. Two rules make them useful: **Describe observable properties, not degrees of goodness.** "Excellent", "good", "fair" restates the number and teaches the judge nothing. "Cites the correct statute but paraphrases the clause" is a test the judge can apply. **Make the boundaries mechanical.** The most valuable sentence in a rubric is usually the one that says what pushes an output *down* a band: "any contradiction of the reference caps the score at 2, regardless of other quality". That gives the judge a rule rather than an impression, and it means a reviewer can check the judge's work. Four or five bands is usually right. Ten bands invites the judge to split hairs it cannot split reliably; two bands is `strict_mode` with extra ceremony. ## What it does and does not buy you It buys **interpretability** — a score maps to a description a human wrote — and **tighter clustering**, because the judge is choosing among described levels rather than picking a real number freehand. It also buys **reviewability**: rubric bands are text in your repo, so a domain expert who does not read Python can audit what the metric rewards. It does not buy determinism. The judge call is still sampled, and a case that sits on a band boundary can land either side of it on a rerun. Bands make that failure mode *legible* — you can look at the boundary case and decide whether your band description is ambiguous — but they do not eliminate it. Whether the resulting scores agree with human judgement at all is a separate question, answered by checking the metric against labelled examples rather than by any argument about configuration. ## How it composes with the other knobs - With **`evaluation_steps`**: steps say *how to look*, the rubric says *how to score what you saw*. They complement each other, and a metric that a pipeline depends on usually has both. - With **`threshold`**: choose the threshold to match a band boundary, so "passing" means a describable level of quality rather than an arbitrary cut. - With **`strict_mode`**: the two pull in opposite directions. Strict mode collapses everything to 1 or 0, which discards the graded scale a rubric exists to create. Use one or the other. ## A worked read Suppose your correctness metric with the rubric above returns 0.6 on a case. Because the bands are defined, you know immediately that the judge placed it in the 3-6 band — broadly correct, missing a required detail — and the `reason` should name the missing detail. That is a bug report. Without the rubric, 0.6 is a mood, and the next conversation is about whether the metric is any good rather than about the output that failed.