You get three numbers on the slide that decides whether an AI red-team programme is funded next year. Which three do you choose, and how do you defend each against the objection that the team producing it can move it?

## The design rule Every metric a programme is funded on gets optimised, so the selection criterion is not accuracy — it is ***who can move the number***. A figure its own producer can raise unilaterally will be discounted the moment a sceptic notices, and the discount is permanent. Pick three whose value is set at least partly outside the team, and say so on the slide. ## Metric one: new causes **1. Confirmed issues whose cause was new to the risk register, banded by severity.** - **Pays for:** reaching a surface or technique that has not been reached before. - **Gaming resistance:** the novelty denominator is the **risk register**, maintained by risk rather than by the red team, and re-running a prolific probe family scores zero here because its causes are already recorded. - **What it costs:** reconciling each confirmed item against the register is roughly 15-30 minutes with a risk owner, so 40 items a quarter is a couple of engineer-days of pure bookkeeping, and it must be funded or it will be skipped. - **How it can still mislead:** the number is set by the register's **granularity** more than by the testing. A register with five coarse causes makes almost everything a repeat and drives the metric to zero; a register with five hundred fine-grained ones makes almost everything new. Publish the register's cause count and its revision alongside the metric, or the trend is uninterpretable. It can also be legitimately near zero in a good quarter on a mature product, which is why it must never appear without the second number. ## Metric two: untested surfaces **2. Named untested surfaces with reason codes, against a dated enumeration.** - **Pays for:** honesty about scope, and it is the ask itself. The **reason codes** split the list into: - a funding request (budget exhausted, with a measured cost per attempt behind it), - an engineering request (no harness), - a third-party negotiation (not authorised) - and a recorded decision (out of scope by design). - **Gaming resistance:** shrinking the list requires actually testing something, and the **enumeration** is owned by architecture. - **What it costs:** a standing review with architecture per product per quarter to keep the enumeration current. - **How it can still mislead:** the list shrinks if the enumeration shrinks, and a surface nobody wrote down is invisible in both directions — so publish the enumeration's revision date and its diff, not just the ratio. ## Metric three: unowned items **3. Unowned open items — count and maximum age.** - **Pays for:** forcing decisions. An item nobody is fixing and nobody has formally accepted is the **escalation** that the reporting exists to produce. - **Gaming resistance:** the red team cannot lower it by working harder; only an owner fixing, mitigating or explicitly accepting moves it, and that **asymmetry** is precisely what makes it credible under challenge. - **What it costs:** every item needs an ownership field and a decision SLA, which is process work someone must maintain. - **How it can still mislead:** it also falls when someone bulk-accepts a batch of risks, so an improvement by paperwork looks identical to an improvement by engineering. Report acceptances as their own line so the two are distinguishable. It is also politically loaded — it makes other leaders look bad in a shared room — so it has to be introduced as an **estate measure**, agreed with those owners in advance, or the programme loses the cooperation it depends on. ## What goes underneath, never on top Attempts run, query budget consumed and cost per attempt belong on the slide as **spend and context**. They explain the shape of the work — why a single-turn sweep is cheap per attempt and an agent engagement is not — without ever becoming the target, because as a target they pay for volume, and volume is the one thing the programme most needs not to be paid for. ## What I deliberately leave off, and why - **Raw scanner hit counts**, because they move with catalogue and sampling configuration rather than with the system. - **Total confirmed findings**, because it is a production count and production counts reward the noisiest instrument. - **A remediation mean**, because its clock runs on teams the programme does not control and it improves when the team files fewer hard findings. ## The caveat to say out loud, before someone else does None of the three measures **residual risk**, because nothing a red team runs can. They measure: - where the instrument was pointed, - where it was never pointed, - and what nobody has decided about. Claiming more than that is the fastest way to lose the argument the first time a real incident lands on a surface the sheet implied was covered — and that argument, once lost, takes the funding with it.