You own the severity rubric for AI findings across an organisation where every control the findings target fails some fraction of the time. How do you let a measured success rate into the rating, and what do you refuse to let it do?

The purpose of an organisation-wide rubric is not precision. It is a *defensible and comparable ordering* of a fix queue produced by different people, using different tools, against targets that all fail some fraction of the time. Every design choice below follows from that goal. ### The rules that carry the weight 1. **Impact is scored from a single success and is never probability-weighted.** Write the ban into the rubric text, not just into training, because the multiplication is intuitive and will otherwise reappear every quarter. Define impact levels by *outcome*: cross-tenant data, regulated or dangerous content delivered to a user, an action taken on a user's behalf, reputational-only, cosmetic. 2. **The rate enters exploitability only, as the last and weakest term.** Reachability comes first — who can reach the entry point, authenticated or not — then retry cost, then the measured rate, capped so it can move the band by at most one step and only in conjunction with a retry-limiting control. 3. **A floor.** Any reproducible success at a user-reachable entry point cannot fall below a defined minimum, however low the rate. 4. **Mandatory metadata on any filed rate:** successes over attempts (not a bare percentage), the success criterion and who validated it, the measurement point, the date, and the compensating controls the rating assumes. A rate lacking them does not enter the score at all. ### The incentives you are designing against - **Sample shopping.** If the rate lowers the score, someone will run twenty attempts and file the number that suits their argument. Mandatory counts plus the floor remove the payoff. - **Judge shopping.** The cheapest lever of all, and invisible without rule 4: swap to a stricter detector or grader and the same transcripts yield a lower rate, with no dishonesty required and often no awareness that anything happened. - **Demonstrability bias.** A finding that reproduces on demand feels severe. Make reproducibility a confidence tag on the evidence, explicitly not a score input. - **Stale scores.** Both the model and its wrapper move. Every rating built on a measured rate carries an as-of date and a re-measure obligation before the finding may be closed as fixed. ### What the rubric costs the organisation Each mandatory field is time on every finding, and the bill is not evenly distributed. Counts are free — the tool already has them. A declared measurement point and a date are free. **Judge validation is the expensive one**: a stratified human read of transcripts, per configuration, an hour or two of senior time each. So scope it — require validation only where the rate was load-bearing, meaning it actually moved a band; elsewhere require only that the criterion be *named*. Coarse bands cost you resolution and buy you fewer arguments, and that trade is right whenever samples are small, which on real engagement budgets is always. Calibration reviews cost a couple of hours a quarter and are the only way you learn whether two teams produce the same answer. ### Where a rubric's own numbers mislead A weighted formula that outputs "6.8" implies a precision none of its inputs have: a rate off a 20-attempt run enters the formula and leaves as a decimal that no one can question, because arguing with a decimal feels like arguing with arithmetic. Prefer small ordinal bands over computed scores. And treat cross-team comparability as a claim to be tested rather than an achievement of publishing the document — have two teams rate the same three findings blind and compare, because a rubric everyone reads differently is worse than an explicit convention everyone knows is rough. ### What to accept You will not get statistical rigour on real engagement budgets, and chasing it burns hours better spent widening what was tested. Design for robustness to weak samples instead: the rubric should produce the same ordering whether the tester ran 20 attempts or 200, with only the exploitability nudge differing. ### How you know it works Sample closed findings and ask whether the ordering still looks right in hindsight. Look specifically for inversions — a reliable-but-cosmetic item fixed before a rare disclosure. Count how often a severity was argued down after a rerun, and check whether that rerun was *larger* or *smaller* than the original; smaller is sample shopping and should be treated as a process defect. And verify that findings whose rating named a compensating control were actually re-rated when that control changed, because a dependency nobody revisits is a rating that quietly expired.