An AI red-team programme's dashboard leads with mean time to remediate, but the fixes are made by application teams and by a model provider the programme does not control. What breaks about that metric, and what would you put on the dashboard instead?

## What the borrowed metric assumes **Mean time to remediate** comes from vulnerability management, where it works because three things hold, with the clock starting at disclosure and stopping at deploy: - each finding has **one owner**, - **one discrete fix**, - and an **unambiguous closed state**. Applied to AI red-team findings, none of the three holds, and each break moves the number in a way that has nothing to do with the testing team's performance. ## Break one: the clock runs on someone else's queue A behaviour can be fixed: - in the application's own system prompt or retrieval layer, - in a guardrail's threshold or category configuration, - in the choice of model version pinned by the client, - or nowhere the organisation controls at all — because the behaviour is a property of a hosted model whose vendor roadmap is invisible. The programme owns **discovery** and nothing after it. A quarter where the mean doubles may mean one application team was reorganised, or that the vendor released nothing that quarter. ## Break two: there is no closure event A **probabilistic control** does not go to zero. A team ships a filter and the reproduction rate falls from roughly 40% of attempts to something small but non-zero. Is that closed? Without a stated **exit criterion** the ticket closes on an opinion and reopens the next time anyone tries the prompt. The defensible criterion is a measured reproduction rate below a threshold agreed *before* the rerun, on the same attack set at the same sample count, recorded as **mitigated with its residual rate** rather than as fixed. ## Break three: verification costs the budget again, and rare things are expensive to measure If the original attack set was 300 prompts at 10 samples, every "is it fixed yet?" is another 3,000 calls, and a three-iteration fix cycle spends it three times. Worse, the statistics of rare events bite exactly where mitigations land: - At a true 2% reproduction rate, a 20-attempt rerun returns zero hits about two-thirds of the time, so "we could not reproduce it" is the expected outcome of an **under-powered check** even when the behaviour is entirely present. - Distinguishing 2% from 0% with any confidence takes hundreds of attempts. - Estimating a 2% rate to within about a percentage point takes on the order of 750. That cost lands on the testing team's calendar while the metric it feeds scores somebody else. ## How the mean itself misleads The distribution is **bimodal**: a pile of trivial items closed same-day and a long tail of accepted or unowned ones, so the mean describes no real item and moves mostly with the mix. Three specific misreadings follow: 1. A **silent provider model swap** can make a batch of behaviours stop reproducing, closing items and improving the mean with nobody having fixed anything — and it can as easily reintroduce them next month. 2. **Accepted risks** sitting in the same pool either inflate the mean forever or get quietly excluded, and which one is happening is rarely written down. 3. And the ugliest incentive of all: the mean improves when the team files fewer hard findings, so a metric meant to reward remediation quietly rewards **shallower discovery**. ## What to put up instead — two groups, clearly separated *Owned by the programme, reported as performance:* - median time from hit to **triage decision**; - median time from triage to a notification carrying a working reproduction and its measured rate; - **verification turnaround** once a team declares a fix ready; - and the **precision of the automated stage**, because it explains the triage load behind the first two. *Owned elsewhere, reported as estate state rather than as score:* an ageing view of open items by status — - fixed and verified by rerun, - mitigated behind a compensating control with a residual rate, - formally accepted with a named accepter, - and **unowned**. The unowned bucket is the point of the whole view. An item nobody is fixing and nobody has accepted is an **escalation**, and attaching a count and a maximum age to it is how a testing team forces that decision without pretending to manage another team's backlog. ## What to check 1. Pull the ten most recent closures and ask for the **rerun evidence** on each: attack set, sample count, measured residual rate, date. Closures justified by "no longer reproduces" with no mitigation shipped are sampling luck or a vendor-side change, and both will come back. 2. Then look at the **histogram** rather than the mean. 3. And confirm accepted-risk items are excluded from the timing pool and reported separately. A remediation clock with no defined stop is not a metric; it is a negotiation, and it will be renegotiated in the room where it hurts most.