Why does reporting raw threat count as a threat modeling metric backfire?

Raw threat count - the number of threats a team enumerated - is the metric a new programme reaches for first, because it is trivially available and it goes up. It is also the fastest way to teach an organisation to produce worse threat models. ### The unit is not a unit Threat granularity is an analyst's choice, not a property of the system. *An attacker replays a captured session token* can be written as one threat, or split into capture on the wire, capture from logs, capture from a browser extension, and reuse after logout. Both descriptions can be defensible analysis. That means the metric has no fixed unit: two teams doing equally good work can report numbers that differ by a factor of five, and one team can change its number without changing its analysis at all. Any metric whose unit is set by the person being measured is not comparable across teams and not stable over time. ### Making it a target completes the failure When a media company began reporting each team's raw threat count to the CTO monthly, teams learned within two cycles to split one threat into nine entries. Nothing about the systems changed; the reports improved. This is the effect usually summarised as Goodhart's law: once a measure becomes a target, the cheapest way to move it is to change the measuring, not the thing measured. The damage is not only cosmetic. Inflated splitting has real costs: - Triage load rises with no increase in information, so the serious findings are diluted among near-duplicates. - Follow-through metrics decay, because closure rate and mean time to close are now computed over a population padded with trivia. - Analysts learn that volume is the deliverable, which is the opposite of the judgment the practice needs. - Cross-team comparison becomes noise, and the teams that resisted splitting look worst. ### The direction of the number is ambiguous anyway Even unmanipulated, a threat count cannot be read. A low count may mean a genuinely simple and well-segmented design, a session that was rushed, or a scope drawn too narrowly. A high count may mean a rich attack surface, a scope drawn too wide, or an analyst who enumerates variants. A metric that cannot tell good news from bad news without opening the model is not a metric; it is a prompt to go and read the model. ### The mirror-image gaming Counting *open* threats instead of found threats inverts the incentive but not the problem: the cheapest way to reduce open threats is to close them as accepted risk without changing anything. Any count used as a target will be met by the least expensive available behaviour. If you count open findings, you must sample how findings were closed, or you have simply moved the gaming downstream. ### What to report instead Report the numbers whose cheapest path to improvement is the behaviour you actually want: | Instead of | Report | Because the cheapest way to move it is | | --- | --- | --- | | Threats found | Coverage of tier-1 designs, with a stated denominator | Model something that was not modeled | | Threats found | High-severity findings closed with a linked change | Ship a fix | | Open threats | Mean time to close, per severity | Close things sooner, not looser | | Any count | Escaped issues and what each taught the practice | Actually miss less | Add the input no counter provides: read a sample of models each quarter and judge whether the threats found are the ones a competent attacker with the assumed position would choose. A programme with modest counts and sharp models is in far better shape than the reverse. ### When a count is still useful Volume is legitimate as an internal diagnostic, never as a target and never in a league table. Forty threats out of one session usually means the scope was too large to reason about. Three threats on a service that fronts the internet and holds customer records usually means the session was shallow or the boundaries were drawn too tightly. Used that way the number is a trigger to look, which is the only role it can honestly play. ### Saying this to an executive The argument that lands is not *the metric is philosophically flawed*; it is that the number they are being shown is set by the people reporting it, and it will improve regardless of whether the systems do. Offer the replacement in the same breath: coverage of the systems that matter, closure of the severe findings, and how many design issues escaped. Those move only when the estate moves.