A fixed-attack jailbreak benchmark like JailbreakBench ships a harm judge that labels each model response as jailbroken or not. Why is that judge part of the frozen artefact, and what breaks in your reported numbers if you substitute your own judge?

**The judge is a measuring device, and it has its own error profile.** A jailbreak attack-success rate is a ratio. The frozen attack set fixes the denominator: how many attempts, against how many behaviours, at what attempt budget. The judge fixes the numerator: which of the resulting responses are counted as hits. That is not a supporting role. It is half of the number, and it is the half that involves a fallible classifier reading ambiguous text. Every harm judge has two error rates. Its **false-positive rate** is how often it labels a refusal, a lecture, or a harmless generic answer as a jailbreak. Its **false-negative rate** is how often it misses a genuinely compliant answer — typically one wrapped in fiction, one written in another language, or one that opens with a refusal paragraph and then complies in the third. Both rates flow straight into the headline. A judge five points more permissive than another shifts *every* entry it scores by roughly five points, which is frequently larger than the gap between the two defences you were trying to distinguish in the first place. **Why the suite freezes it rather than leaving it to you** Without a frozen judge a leaderboard is not a leaderboard. Entries labelled by different graders sit on different scales, and there is no way to tell a genuinely stronger defence from a stricter grader. Freezing the judge is exactly as load-bearing as freezing the prompts, and the standard mistake is to treat the prompt set as "the benchmark" and the judge as a utility you may swap for a better one. **What breaks the moment you substitute** Your entry stops being comparable to every published number on that suite, while continuing to look like the same metric with the same name and the same units. That is the dangerous part: nothing errors, nothing warns, and a slide reading "12% on JailbreakBench" is now unfalsifiable unless the judge is named beside it. A stricter in-house classifier will usually push the rate down and make your defence look better than the leaderboard would; a permissive one flatters the attacker. **Where the shipped judge legitimately falls short** It was calibrated for the suite's own harm taxonomy, not your product's. A response that is unremarkable in general can be disastrous inside your domain — a plausible-sounding dosage, a fabricated policy quote, an internal identifier — and the shipped judge will label it clean. It may also be blind to partial compliance. And if the judge is itself a model behind a hosted API, its behaviour can move under you between runs as the provider updates it, which means your supposedly frozen instrument was never fully frozen. **The pattern that keeps both properties** Score every run twice, from the same stored transcripts: ``` stored responses -+-> shipped judge (pinned) -> the comparable, publishable number \-> your product judge -> the number the release decision uses ``` Report them as two labelled columns, never as one number. Store the raw transcripts, not just the labels: with transcripts, a corrected judge, a new judge, or a scoring bug can be re-applied to every historic run offline. **What it costs** Judging is the cheap half if you keep transcripts, and the expensive half if you do not. Re-labelling a few thousand stored responses with a model-backed judge is a handful of dollars and minutes; re-running the attacks to get those responses back is the full benchmark bill again against a metered endpoint, plus the risk that the endpoint has changed underneath you so the new run is not the old run. Hand-labelling is the genuinely expensive item: a stratified sample of 100–200 responses is several hours of an experienced reviewer, and it needs a second reviewer on a subset if you want to quote human agreement rather than one person's opinion. **Where the number misleads** Two failure modes dominate. First, a judge-driven delta read as a defence-driven delta — you tightened the grader between quarters and the improvement is entirely artefact. Second, a real but unresolvable gap: if defence A and defence B are separated by two points and the judge disagrees with human labels on five points' worth of borderline responses, you have not measured a difference, you have measured the grader's ambiguity band. **What to check** That the shipped judge ran unmodified and its version is recorded; that transcripts are stored, not just labels; that a stratified hand-label sample — clear refusals, clear hits, and heavily weighted to the borderline band — has been done recently; and that judge-human agreement on that band is reported next to any gap you are about to act on.