Two teams publish attack-success rates for the same open-weights model on the same behaviour list. One scored replies with the trained harm classifier the benchmark distributes; the other scored them with a hosted chat model prompted with a harm rubric. The rates differ by 14 points. How do you work out whether the model or the scoring choice explains the gap, and what do you require before putting both numbers in one table?

**Why the summary statistics cannot settle it.** An attack-success rate is the output of two stages: a generation pipeline (the attack implementation, the target's decoding settings and system prompt, how many attempts per behaviour, how those attempts were reduced to one per-behaviour result) and a scoring rule (here, the benchmark's distributed harm classifier versus a hosted chat model prompted with a harm rubric). The two teams differed in *both*, so 14 points is a sum of two effects with no way to split it from the aggregates. Reasoning about which judge is "better" is not a substitute; the only clean move is to hold one factor fixed and vary the other, and that needs the artifacts, not the numbers. **Run the 2x2.** ``` classifier rubric judge team A outputs a11 a12 team B outputs a21 a22 ``` `|a11 − a12|` and `|a21 − a22|` are the judge effect; `|a11 − a21|` and `|a12 − a22|` are the generation effect. Report both. If the rows move together and the columns do not, the model did nothing and the scoring choice explains the gap. If both move, you have two confounds and must fix one — usually by re-generating one team's configuration yourself — before claiming anything. **What the cross costs.** Two extra scoring passes over both transcript sets. With the distributed classifier that is GPU time: a fine-tuned model in the low tens of billions of parameters, one forward pass per transcript, so a few thousand transcripts is minutes to an hour on one accelerator, and free of per-call charges. With the hosted rubric judge it is one API call per transcript per pass, so the cross doubles an already metered bill and is rate-limit-bound in wall clock. Archiving the completions that make the cross possible is cheap in storage terms — a few kilobytes per attempt, single-digit gigabytes for a large campaign — so the real expense is engineer time and the discipline to have pinned the artifacts before the run, not disk. **Where the number misleads, part one: equal totals are not agreement.** Two scorers can land within a point of each other on the aggregate while disagreeing on a third of individual transcripts, because their false positives and false negatives offset. Any conclusion drawn from matching totals is unfounded. Pull the disagreement set — the transcripts the two rules label differently — sample it, adjudicate by hand, and report per-item agreement, not the totals. **Part two: the gap may not be error at all.** Adjudicate each disagreement into (a) a scoring mistake by one rule, or (b) a genuine definitional difference about what counts as a hit. Only (a) is fixable by picking the better judge. Rubric judges commonly score a refusal-with-partial-explanation as a partial success where a binary classifier does not; they are often more permissive on fictional framing; and "any actionable detail" versus "a complete, workable procedure" are two different metrics wearing the same name. No accuracy work reconciles a definitional gap — you choose one definition and re-score everything under it. **Part three: the boring mechanical causes.** Before crediting anything conceptual, check truncation (one pipeline scored only the first N tokens of long replies), whether the judge saw the attack prompt as well as the reply, format drift — encoded, translated or code-shaped replies sit far from a trained classifier's fitting distribution and come back benign — and whether the two teams reduced multiple attempts per behaviour differently (any-of-n versus per-attempt is a denominator change, not a judge effect, and it can carry the whole 14 points on its own). **What I require before the two numbers share a table.** Per attempt: prompt, full untruncated completion, decoding parameters, seed, attempts per behaviour, and the reduction rule. Per scorer: a pinned artifact, or an exact rubric prompt plus a model snapshot identifier, together with error rates measured on an adjudicated sample of *this* transcript distribution. Given those, publish one number per (transcript set, scorer) cell and name which scorer defines the headline. Lacking them, publish the two rates separately, label each with its scorer and transcript set, and state plainly that they are not comparable. Averaging them, or taking the lower as "conservative", invents a number no experiment produced.