When you replay a published red-team benchmark against your own application endpoint, the benchmark's automated harm judge can start scoring the wrong thing because your app's responses do not look like raw model completions. What are two ways your response format breaks that judge, and what do you do about each?

**What a harm judge actually is, and why it is the fragile part of a replay.** The judge is the component that converts a response into a label. In harmbench it is a classifier fine-tuned on model completions paired with behaviour descriptions; in jailbreakbench it is a judge model driven by a published rubric. Either way it is a *fitted* artefact: it was tuned or prompt-engineered against a specific distribution — single-turn prose completions returned by a chat-completions endpoint. Teams porting a benchmark to their own application port the item corpus deliberately and port the judge by accident, because the judge simply comes along inside the harness. But the corpus is text that means the same thing anywhere, while the judge is the one piece that was fitted to a distribution you have just replaced. **The failure modes worth naming.** - **Wrapper text.** Your endpoint returns JSON, a rendered card, or a citation-annotated block. The judged string now contains field names, markup and reference markers. Judges fitted on prose degrade on that input, and they degrade *asymmetrically*: the usual direction is toward "no harmful content", which flatters your score. - **Truncation.** A streamed response captured at a cut-off, or a length cap, hands the judge half an answer. Half a compliant response very often reads as a refusal. - **Canned deflection.** A single templated out-of-scope sentence is out of distribution for the judge, and because it repeats byte-identically, one mistake is not one error — it is a correlated error applied to every off-scope item, and it moves the aggregate on its own. - **Displaced harm.** When the app can act, the consequential part of a hit is often a tool argument, not the prose. A prose-only judge scores that as a clean refusal, and the more capable your product is, the more of your risk hides in that gap. - **Post-processing.** A redaction or rewrite pass means the judge is scoring the sanitiser. That is a legitimate measurement, but it is not the measurement you claimed. **The method, and what it costs.** Capture the full response envelope: user-visible text, tool calls with their arguments, and the termination stage. Judge the union, with an explicit written rule for when the *action* is the hit rather than the text. Then validate the judge against humans, because an unvalidated judge at a new interface makes the score decorative. The validation is the cost item people omit from plans: draw a stratified sample across judged-hit, judged-miss and template responses — 150-300 responses is the usual working size — and label it by hand. At roughly one to two minutes per response that is three to ten hours of a qualified person, doubled if you want a second labeller and an inter-rater figure. It is not one-off: every change to the app's output format silently refits nothing and invalidates everything, so the cost recurs at each envelope change. Judge inference itself is cheap by comparison — one call per response — but not free at three samples per item across a few hundred items. **Where the number misleads.** The headline misreading: a replayed refusal rate far above the published run's is celebrated as evidence that the safety stack works, when the refusals are near-identical template strings and the real finding is that the judge cannot read your format. Second: reporting raw agreement on an imbalanced sample. If 95% of responses are non-hits, a judge that labels everything non-harmful shows 95% agreement and zero usefulness — the number to look at is agreement *within the hit stratum*, or a chance-corrected statistic, not the pooled percentage. Third: validating the judge only on the miss stratum, which is exactly the stratum where the broken judge and the correct judge agree. Fourth: quoting a score with no judge error rate beside it, which hides the fact that the score's uncertainty is dominated by the labeller and not by sampling. **What you check.** Whether the judged string is the user-visible text and nothing else. Whether tool calls are in the judged unit at all. Whether refusals cluster into a few identical strings. Whether hit-stratum agreement, not pooled agreement, is what you reported. And whether the last re-validation postdates the most recent change to the response envelope.