Harmful-Behaviour Sets

**What the file actually is.** A harmful-behaviour set is a data file and nothing more: a few hundred rows, each row a short description of something you do not want a model to do, plus a category tag its authors chose. AdvBench, released alongside the GCG optimisation paper, is a list of that shape — its harmful-behaviours split runs to roughly five hundred rows of one-line requests. HarmBench is another, a few hundred textual behaviours grouped into semantic categories, shipped with a companion harm classifier and a standardised evaluation harness. Check the row count and the revision of the copy you actually have; both projects have been edited since publication. The file attacks nothing and judges nothing. It defines one thing only: the population of behaviours that will be attempted. **Three components wear the same name.** A benchmark result is produced by three separable pieces, and a bare name tells you about one of them. | component | what it is | what changes when you swap it | |---|---|---| | the behaviour list | static rows plus category tags | the population measured; an omitted harm is never attempted at all | | the attack | whatever turns a row into prompts — a direct ask, a role-play template, an optimisation loop, a published transfer suffix | the number, on the same list against the same model | | the judge | the rule that decides a response counted — a refusal-keyword matcher, a fine-tuned classifier, an LLM grader, a human | the number again, over byte-identical transcripts | So “passed HarmBench” is an incomplete sentence. The complete one names the list and its revision, the attack that drove it, the judge that scored it, and the denominator — how many behaviours were attempted, and how many generations per behaviour. **What a run costs.** The direct-ask baseline is cheap: one completion per behaviour per generation, so four hundred behaviours at five generations is two thousand completions — minutes of wall clock and single-digit dollars on a hosted endpoint. Judging roughly doubles the call count if you use an LLM grader; a shipped classifier such as HarmBench's runs locally and costs GPU minutes instead. The expensive tier is optimisation-based attack: a per-behaviour gradient search needs white-box weight access and GPU-hours *per behaviour*, which is hundreds of GPU-hours for a full list, and is why most teams replay published transfer strings at direct-ask price instead of re-running the search. The real bill, though, is human: reading flagged transcripts runs about an hour per fifty to a hundred responses, and that reading — not the aggregate — is where a finding comes from. **Where the number misleads.** Several specific misreadings, each common: - *The direction is unstated.* “96%” can be refusal rate or attack-success rate. Written without the word, a good number and a catastrophic number look identical. - *Omission reads as success.* A harm with no row produced no prompt, so it produced no failure. In the aggregate an unmeasured harm and a refused harm are the same absence. - *The attack was weak.* Direct, unadorned requests sit near the ceiling on any instruction-tuned frontier model, so a near-perfect refusal rate under direct asking is close to a null result; the same list under a strong attack routinely collapses it. A number without its attack cannot be compared with anyone else's. - *The judge scored the wrong thing.* A keyword-based refusal detector marks any response starting “I’m sorry” as a refusal, including one that apologises and then complies, and marks incoherent output as an attacker win. Two teams with the same list and attack diverge on judging alone. - *The list is public and old.* Widely published rows plausibly appear in safety-tuning data, so a high score can reflect exposure to these exact strings rather than generalised refusal. - *The target was the bare model.* A result on the raw API says little about your deployed assistant, which carries a system prompt, retrieval, guardrails and tools — each of which can help or hurt. **What to check before repeating the claim.** Which list, which revision, how many rows. Which attack, and whether it was run or replayed. Which judge, and its agreement with a human on a sampled subset — fifty transcripts is enough to expose a badly calibrated grader. Per-category counts, not just percentages. Whether the target was the model or the product. And finally: has anyone on the team opened the file? If nobody has read the rows, nobody knows what the number covers, and the honest reply to “we passed it” is a request for the denominator.