In your AI red-team harness, whether an attempt counted as a failure was decided by an automated model-based grader with a threshold. The application team will re-run the reproduction without that grader. What do you put in the handoff so their verdict matches yours?

This is where evidence packages quietly fall apart. The transcript travels fine; the *judgment* does not. The team re-reads the same response, sees a hedged, partly-refusing answer, and concludes it is acceptable — because nothing in the package told them what standard was applied. ## What a grader actually is A model-based grader is a second model call. It receives the response (often the prompt and a rubric too) and returns a score, and a threshold turns that continuous score into a binary verdict. The shape recurs across tools: a garak detector returns a per-attempt score in the 0.0–1.0 range and the report's pass/fail follows from it; promptfoo's `llm-rubric` and its other model-graded assertions take a `threshold` in the test's assert block; PyRIT's self-ask scorers return either a true/false or a float on a scale defined inside the scorer's own prompt. In every case there is a prompt you wrote, a judge model with a version, and a cut point you chose. All three are invisible in the verdict, and all three are load-bearing. ## Three layers of handoff **The rule in prose, with its threshold.** "Counts as a failure if the response provides actionable steps toward the objective, scored above 0.7." A human can apply that to a response their fix has newly produced, which is what verification actually requires. **The grader's output on the attached cases.** A score per transcript, so the reader sees the margin. A case that scraped past the cut should be flagged as marginal rather than presented as equivalent to a clear one. Scores without the rule are numbers with no units; the rule without scores hides that some of your evidence was borderline. **A deterministic proxy for the regression test.** Where the failure has an observable, mechanical signature, hand that over and let the model-graded version stay in your harness. Where it has none, say so explicitly and agree what re-verification looks like, rather than letting it default to nothing. ## What it costs Model-graded evaluation roughly doubles the call count of a run — one call to the target, one to the judge — and the judge call is often the more expensive of the two, because its input carries the whole response plus the rubric. A twenty-trial re-verification is forty calls. Put that in the application team's pipeline and a suite of thirty graded assertions is thirty target calls and thirty judge calls on every run, priced per run, against a hosted judge that can reversion without notice. The first time it flakes red on an unchanged pull request it gets marked allowed-to-fail; the second time it is deleted. Engineer time is the larger half of that bill. This is the whole argument for pushing a deterministic assertion into their pipeline and keeping the judge in yours. ## Where the number misleads A score of 0.81 against a 0.7 threshold looks decisive. It is a continuous score whose errors concentrate exactly at the boundary you drew. Model graders carry well-documented systematic biases: longer and more fluent answers score higher; hedged, polite, refusal-shaped text scores lower even when the operative content is present; formatting and ordering shift the read; and a judge drawn from the same model family can favour that family's outputs. The characteristic miss in safety work is the difference between a response that *describes* a hazard and one that operationally *enables* it — a rubric that does not draw that line will flag the first, and that is the attachment the developer will dispute. The aggregate inherits all of it. If your judge agrees with careful human marking on 85% of cases, your measured 35% rate carries an error the report does not display, and those errors are not randomly scattered: they cluster near the threshold, which is precisely where your weakest attachments sit. Push-back always comes on the weakest attachment, never the strongest, so a rate you have not spot-checked is a rate you cannot defend. ## What to check before you send Read every attached response yourself, without looking at the score, and mark it; any attachment where your judgment differs from the harness's verdict comes out of the package or ships with the disagreement noted. Then measure the grader rather than trusting it: hand-mark a blind sample drawn from *both* the flagged and the unflagged responses. Sampling only the flagged ones measures precision and leaves the grader's misses completely invisible. Report the disagreement rate in both directions, along with the judge model and version, so the number the team inherits comes with its error bar attached.