Standard Suites

## A fixed-attack suite is an instrument, not an assessment An **assessment** asks "is this system safe?" and has to consider everything. An **instrument** asks one narrow question and answers it the same way every single time it is used. A fixed-attack jailbreak benchmark — JailbreakBench is the canonical example of the class — is deliberately built as the second thing, and every strength and every weakness it has follows from that one design choice. ## The three artefacts it freezes - **The attack set.** A published, versioned list of attack artefacts, each tied to a **behaviour** — a named harmful capability the attempt is trying to elicit, phrased as a target ("produce instructions for X"). Behaviours are the rows of the experiment; the attack artefacts are the attempts made against each row. A suite of this class typically carries on the order of a hundred behaviours, split across a harm taxonomy, plus a matching set of benign behaviours used to check that a defence is not simply refusing everything. Freezing this means a difference between two runs cannot have come from a different attacker. - **The harm judge.** A fixed decision procedure — a trained classifier, a scripted string rule, or a model prompted in a published, versioned way — that reads the pair (behaviour, model response) and emits one bit: jailbroken or not. This is the component that turns text into a number, and it is emphatically part of the measurement rather than an implementation detail. Two labs scoring the *same* stored responses with different judges will publish different rates. - **The report shape.** Which rate is published, over which denominator, and what is published beside it. Attack-success rate per *attempt* (every attempt is a row), per *behaviour* (a behaviour counts once), and per *behaviour with at least one success out of k attempts* are three different numbers computed from one identical run, and the third rises monotonically as you raise k. Freezing the report shape is what lets two leaderboard entries be lined up on a page at all. ## Why the freeze is what makes two defences comparable Comparison is subtraction, and subtraction is only meaningful when everything except the term you care about is held constant. - Change the prompts and you have measured a different threat. - Change the judge and you have changed the definition of the word "success". - Change the denominator and the percentage has changed units. With all three pinned, the delta between defence A and defence B has exactly one place left to come from: the defence. ## What a run costs Target calls are roughly `behaviours x attack artefacts per behaviour x generations per attempt`. A hundred behaviours with a handful of attack artefacts and one generation each is several hundred to a few thousand calls — on a commodity hosted chat endpoint, single-digit to low-tens of dollars and tens of minutes of wall clock at modest concurrency. A model-backed judge roughly doubles the call count and adds its own bill. None of that is the real cost. The real cost is **engineer time**: writing the target adapter so the suite talks to your *deployed* stack rather than a bare model, pinning the suite revision, and hand-triaging borderline labels. Budget the adapter in days and the run itself in minutes; teams routinely size this backwards and are surprised that the cheap part is the compute. ## Where the number misleads The rate answers "how did this defence do against *these* frozen attempts, under *this* judge, over *this* denominator" — and nothing wider. Four specific misreadings recur. 1. First, quoting it as "our jailbreak rate", which is a claim about all attacks and is not supported. 2. Second, comparing a number computed per-behaviour-with-at-least-one-success at k=10 against someone else's per-attempt number, which is a units error dressed as a comparison. 3. Third, ignoring that the prompts are public, so they are the first strings a vendor blocks by rule or by safety-tuning; the score drifts optimistic over time while looking identical. 4. Fourth, quoting a bare-model figure for a product that adds a system prompt, tools and retrieved content — surfaces the suite never touches. ## What to check before believing a posted number - Which suite and which revision; - whether the shipped judge was used unmodified; - the exact denominator and the attempts per behaviour; - the target configuration, including decoding settings and whether it was the bare model or a deployed stack; - and whether the defence under test was developed while its authors could see those prompts. Absent those five, a posted percentage is a number, not a measurement you can compare yours to.