You have one attack-success rate published by HarmBench and one published by JailbreakBench for the same open-weights model. What has to line up before you can put both figures in a single comparison table, and what usually does not?

A published attack-success rate is the output of a complete instrument: an item list, a way of driving attempts at a target, and a procedure that rules whether a response counted. HarmBench and JailbreakBench are two instruments calibrated to nothing in common, so the gap between their headline numbers is not a measurement of anything about the model. ### What differs, and why every difference alone moves the rate - **Population.** Each suite curates its own behaviour list — different size (JailbreakBench's list is on the order of a hundred behaviours; HarmBench's is several hundred), different harm taxonomy, different difficulty mix. A list weighted towards categories a model was heavily tuned on scores low for population reasons. - **Denominator and aggregation.** One suite may report the fraction of *behaviours* broken by any of n samples; another the fraction of *attempts* ruled harmful. These are different statistics even on identical transcripts. - **Attack and budget.** A rate is always the rate of some method at some number of turns or queries. A stronger or simply longer-running attack lifts the number without touching the model. - **Target configuration.** Raw weights with a bare chat template, versus the same weights with a chat wrapper, a system prompt, or a hosted endpoint's own filters. Also decoding: temperature and top-p change what a resampled attempt produces. - **Ruling.** Each suite ships its own harm judge — a classifier or a prompted judge model with its own rubric. Two judges scoring the same transcripts routinely disagree on a noticeable fraction of borderline cases, and that disagreement lands entirely in the numerator. Five free variables, one observation. Nothing can be attributed. ### The move that actually works Hold the instrument fixed and vary only the thing under comparison: 1. Pick one harness and one ruling procedure. 2. Run both behaviour lists through it against the same target configuration, same decoding settings, same attempts per item, same aggregation rule, same attack budget. 3. Report two numbers labelled by list, with the per-category breakdown beside them. Do not average the two into one figure. The lists have different category mixes, so a pooled rate moves with the relative sizes of the two lists — a purely clerical quantity — rather than with the model. ### What that costs This is not free, and the cost is why teams quote papers instead. Two full sweeps of generations plus judging: on the order of `(|list A| + |list B|) x attempts x methods` target calls and a matching number of judge calls, which for a few hundred behaviours at a couple of dozen samples is tens of thousands of calls per model, hours to a day of wall clock against a rate-limited endpoint, and a repeat of the whole thing for every model in the comparison. The larger cost is usually engineer time: the two lists arrive in different schemas with different harm taxonomies and different notions of what a target response is, and normalising them into one harness so that "the same judge" really is the same judge is days of work, not an afternoon. Budget for it once and reuse it; budget for it per-report and it never happens. ### Where the number misleads The seductive reading is that the higher rate is the more rigorous or more honest measurement — that one suite is "harder". It may simply have an easier list scored by a more permissive judge with more attempts per item. There is no shared scale on which one suite over-reports relative to another, so the difference has no sign you can interpret and certainly no magnitude. Two further traps: a leaderboard that ranks models by pooling across suites inherits every one of these confounds at once; and even one suite's numbers from two dates are not automatically one series, because suites revise items and update judges between releases. Public lists also age — once a list is widely mirrored, later models may effectively have seen it, and the rate drifts for reasons unrelated to defences. ### What you would check Before drawing any bar chart: confirm both figures state their aggregation rule and attempts per item; confirm both targeted the same kind of object (bare weights versus a wrapped endpoint); take a couple of hundred transcripts and score them with *both* suites' judges to measure the disagreement rate directly, because that number sets how much of the gap is judging rather than model. If the disagreement is material, the comparison is dead on arrival and the honest deliverable is two labelled series plus the per-category breakdown, which is what a defence can act on anyway.