Measuring the Practice

A threat modeling practice is a **process**, not an artifact, so the honest question a lead or an executive asks is whether it changes the systems being built. Before the metrics, fix the vocabulary they measure: a **threat** is what could go wrong (an attacker abusing a flow), a **vulnerability** is the concrete flaw that lets it happen, a **risk** is the rated consequence of that threat given the system as it stands, and a **control** is what you do about it. A metric that counts threats measures words produced; a metric that counts closed findings measures systems changed. Four families of measurement cover a practice, and each exposes a different failure. ### 1. Coverage - how far the practice reaches The share of in-scope designs or services that have a current model. It is a *leading* indicator: it tells you whether the practice is being applied at all, long before any outcome shows up. Coverage is only meaningful with a stated denominator (which population of services counts) and a stated definition of *modeled* (a diagram exists, versus threats enumerated, rated and owned). Report it weighted by criticality as well as flat, because the unmodeled remainder is where the interesting question lives. ### 2. Follow-through - findings closed and mean time to close Coverage without closure means the practice produces documents. Track, per severity band, how many findings from models are closed with a linked change, and the mean elapsed time from a finding being written to that change shipping. This is the number that proves modeling output reaches the codebase. Two cautions: closure by silent risk acceptance is not closure, so sample how findings were closed, and a closure rate collected only over findings a team chose to record is self-selecting. ### 3. Latency - time-to-model Elapsed time from the trigger (a design review requested, a new service scaffolded) to a finished model with owned findings. This is the number that predicts whether teams will route around you. It only becomes useful when you **decompose** it into requested -> scheduled, scheduled -> session held, session -> findings written and owned. A fintech that measured eleven elapsed days of time-to-model found that nine of them were queue: teams waited a week and a half for a slot with a facilitator, and the analysis itself took under two days. The headline number does not say *modeling is slow*; it says *facilitation capacity is the constraint*. The remedies differ completely - more trained facilitators or a self-serve path for low-risk changes, versus a lighter method if the analysis itself were the slow part. ### 4. Escapes - issues found after the design passed An **escaped issue** is a design-level security problem discovered in production, in a later test, or in an incident, on a system that had been modeled. This is the only genuinely *lagging outcome* measure of the four, and it is slow, sparse and noisy: a quarter with zero escapes usually means nobody looked, not that the practice is perfect. Track it as a rate over modeled systems and, more usefully, review each escape individually. ### Reading them together | Metric | Type | Fails when it says | | --- | --- | --- | | Coverage | Leading | Practice is not reaching real designs | | Findings closed / mean time to close | Leading | Output never becomes change | | Time-to-model | Leading | Teams will route around the practice | | Escaped issues | Lagging | The method or its scope missed something real | Coverage plus latency describe whether the practice runs; closure plus escapes describe whether it works. Reporting only the first pair is the common trap - it makes an activity look like an outcome. ### Instrumentation None of this needs a measurement programme of its own. Timestamps on the model record (requested, session held, signed off), a link from every finding to the ticket that fixes it, and a small quarterly read of a sample of models for quality give you all four. Add one qualitative input: read five or six models a quarter and judge whether the threats found are the ones a competent attacker would pick, because no counter can tell you that. ### What not to put on the dashboard Raw threat counts, sessions held, hours spent, models written, and any per-team league table built from those. They measure activity, they are trivially inflated, and threat granularity is an analyst choice so the units are not comparable between teams. Use them at most as internal diagnostics - a session that produced forty threats usually means the scope was too big, not that the system is forty times worse. A defensible executive report is three numbers and a sentence: coverage of tier-1 services, closure rate and mean time to close for high-severity findings, and escapes this period, followed by what the escapes taught you about the practice.