You lead red teaming for a company shipping a code assistant, a bank chat assistant and a medical triage bot. Would you make a standard harmful-behaviour set such as HarmBench the organisation's unit of measurement, keep a per-product behaviour register, or both? Which would you choose, and what is each number allowed to justify?

**The judgement is about authority, not about which list is better.** Run both, and write down what each number is permitted to prove. Everything else follows from that one decision. **The shared list.** Its single virtue is comparability: the same rows, the same attack, the same judge, across three products and across candidate models. That makes it the right instrument for two questions — “has this model version regressed on generic harms since the last one” and “of these two candidate models, which starts from a better place”. Its cost is that it is product-agnostic by construction, so it can never be evidence that a product is safe to ship. One central team maintains it, one revision is pinned per quarter, and its result is labelled a regression signal on the page where it appears. **The product registers.** Each product's realistic worst outcome is specific: the code assistant proposing a dependency or a snippet that quietly weakens the caller's security posture, the bank assistant resolving the wrong customer identity and disclosing their data, the triage bot giving dangerous advice or failing to escalate a red-flag presentation. None of these is in a generic list. Two of the three are only reachable through the product's own tools and several turns of conversation, and the triage failure needs clinical review to judge at all — no shipped classifier can decide whether an escalation was owed. These behaviours are written locally from the threat model, and a release decision rests on them. **What this costs, concretely.** The shared list is the cheap half: pinning a revision, running it per model candidate and per release, and reading the failures is a few days per quarter for one owner, plus a small compute bill — a few hundred rows at a handful of generations, plus judging, is dollars. The registers are the expensive half and the cost is people: an hour or two to author each behaviour with its target configuration and judging rule, fifteen to twenty behaviours per product, a refresh whenever a product gains a tool or a data source, and for the triage bot a clinician's time to adjudicate outcomes — which is the real budget line and the one that gets cut first. Size it honestly up front: the top handful of risks per product get behaviours, and you say out loud that risks six and below are unmeasured this quarter. **Second-order effects to plan for.** - *Goodhart.* The moment the shared figure appears in a launch deck or a performance review it becomes a target, and the cheapest way to move it is to work on generic harms that were never the product's risk. Keep it explicitly stripped of ship authority. - *Over-refusal.* Scoring only refusal of harmful behaviours rewards refusing, so a triage bot drifts toward declining legitimate clinical questions and the measurement applauds. Each product needs a benign counterpart set — requests that must be answered — reported beside the harmful one, or the trade is invisible. - *Divergence.* Three registers drift in quality within two quarters. A shared entry template, a common report shape and central review of additions and deletions hold them together while the content stays with the team that owns the threat model. **Where the numbers mislead.** A blended company-wide safety score is the classic mistake: it is comparable to nothing external, it hides which population moved, and it lets a strong shared-list result mask a weak register. A per-product register with twenty behaviours produces small counts, so quarter-to-quarter movement is mostly noise — report counts and read transcripts rather than charting a rate. And a rising shared-list number across a model migration can simply mean the new model was tuned on those public rows. **How I would report it, and what I would check.** One page per product: shared-list block (name, revision, rows attempted, per-category counts, attack, judge), register block (revision, behaviour count, per-category counts, target configuration including which tools were live), and one line stating what the shared list does *not* include for this product. The check I would run on any team claiming coverage is to ask the product owner to name the register entry that covers their top risk and show me the transcript of it failing at least once historically. If they cannot, the register is decoration.