Another team publishes "2% attack success" on the same public harmful-behaviour list you used, where you measured 30%, and their write-up never states attempts per behaviour. What can you legitimately conclude, and what do you do to make the two comparable?

**Read their number charitably first.** Under any-hit aggregation the reported rate is monotone in the attempt budget, so an unlabelled figure carries exactly one honest reading: *at whatever budget they bought, that fraction of behaviours fell.* It is a lower bound on what the same run would report with more attempts. Their 2% and your 30% are therefore not contradictory. They are entirely compatible with one model, one list, and two budgets — 2% at n = 1 and 30% at n = 20 are ordinary readings of the same system. Announcing a fifteenfold robustness advantage on that evidence is precisely the error being probed. **The axes that must match before a comparison means anything.** | axis | why it moves the rate | |---|---| | attempts per behaviour | any-hit aggregation is monotone in n; this is the axis their write-up omitted | | the hit rule | a refusal-string check, a trained classifier and a judge model disagree sharply on borderline completions; a swap can move the rate by tens of points with n untouched | | the attack | a single-turn template and an adaptive multi-turn strategy are different experiments even on identical behaviours | | the target's surrounding configuration | system prompt, decoding temperature, and any input or output filter in front of the model; a bare endpoint and the shipped stack are different systems | | the list version | public lists get revised and subsetted; "the same list" is a claim to verify, not assume | The order matters when you investigate. Attempts is the most likely explanation and the cheapest to obtain, but "they used fewer attempts" is a hypothesis you test, not a conclusion you publish. A refusal-string hit rule that counts any non-refusal as success will typically report *higher* than a judge model that demands the completion actually be harmful and on-topic — so a difference in the other direction can also be a hit-rule artefact. **What it costs to settle.** Asking is nearly free and usually works: attempts per behaviour, stopping rule, hit rule, attack, target configuration is a five-line reply, and omitting them is overwhelmingly a reporting-convention failure rather than manipulation. Reproducing is not free. Re-running a documented configuration on a few-hundred-behaviour list at n = 20 is thousands of target generations plus the matching judge calls, hours of wall clock against a rate-limited endpoint, and a day or two of engineer time to wire up their harness and reconcile their hit rule with yours. Budget it deliberately, because the value of the exercise is a comparable table, not the satisfaction of winning an argument. **The move.** Ask for the missing parameters. Failing that, reproduce as far as their setup is documented, at your own declared n, on the verified list version, with your own hit rule, and publish a table whose every row carries list and version, attempts per behaviour, hit rule, attack and target configuration. Because you kept per-attempt outcomes, also present *your* rate at n = 1 — the value they most likely used — which makes the comparison as favourable to them as your evidence permits and removes the suspicion that you picked a budget that flattered you. Where their configuration is undocumented, say so plainly: an unreproducible figure is a citation, not a measurement, and belongs in prose rather than in the comparison table. **Where this misleads if you get it wrong.** Copying their number into your table under your column headings silently asserts it was measured under your conditions, and every downstream reader will treat it that way. Averaging the two produces a figure describing no run at all. Concluding "the model changed" imports a cause the data cannot distinguish from four other causes. And treating the omission as bad faith is both usually wrong and strategically bad — the durable fix is a caption standard both write-ups adopt, not an adjudication of this one dispute. **What you would check.** That the list you both used is the same version and the same subset. That their stopping rule is not early-stop-on-first-hit, which changes what per-attempt statistics you can recover from their logs. That the hit rules are stated well enough to be re-implemented. That their target was measured bare or behind a filter, and which one yours was. And whether their write-up reports any figure at more than one n — if it does, the slope tells you where on the saturation curve their single headline number sits.