Your dashboard reports 94% of services threat-modeled: how do you check that number is honest?

A coverage percentage is the easiest threat modeling metric to produce and the easiest to fool yourself with, because the number carries none of the three choices that determine what it means: the population being counted, the test for being counted, and how recently the test was passed. ### Interrogate the denominator first *94% of what?* In the payments processor case, the denominator was every service with a repository in the company's primary source-control organisation. That sounded like the whole estate and was not. Outside it sat an acquired subsidiary that still ran its own build and hosting, a handful of vendor-hosted integrations that process the same cardholder data, internal admin tools nobody thought of as services, and scheduled jobs that move data between them. Measured against the full inventory, the same numerator gives a much lower figure. The general failure is that the denominator is defined by *where the measurement was convenient to collect* rather than by what is in scope. Fix it by reconciling against an independent inventory that was built for another purpose - the asset or service catalogue, cloud accounts and their billing lines, the DNS zone, the list of systems in scope for the last compliance exercise. Wherever those disagree with your denominator, you have found either an omission or a legitimate exclusion; write down which, so the exclusions are visible rather than silent. ### Then interrogate the numerator *What earns a tick?* Common definitions, in increasing strength: 1. A diagram exists somewhere for the service. 2. A diagram exists and threats were enumerated against its elements and flows. 3. Threats were enumerated, rated, and each accepted or paired with a control and an owner. 4. All of the above, and the controls have shipped or have dated tickets. Definition 1 is what most inflated coverage numbers use, and it counts artifacts rather than analysis. A service can hold a diagram drawn eighteen months ago by someone who has left and still count. Say which definition you are reporting; if you cannot, the percentage is not a measurement. ### Then freshness A model is a claim about a system as it was on a date. Coverage should count a model as current only if it is newer than the last significant architecture change to that service - a new external integration, a new datastore, a new trust boundary, a change of authentication. Coverage without a freshness rule slowly converts into a count of historical documents while the estate moves underneath it. ### Then weighting Ask what is in the missing 6%. A flat percentage treats a tier-1 payment authorisation path and an internal dashboard as one unit each, so a programme can hold a beautiful headline number while every system that would actually hurt you sits in the remainder. Report criticality-weighted coverage alongside the flat figure, or better, report the tiers separately: coverage of tier-1 services is the number that deserves the slide. ### How to check the claim in practice - Pull the denominator from a source your practice does not own, and diff it against your list. - Take a random sample of ten services counted as covered, open their models, and see which definition they actually satisfy. - Check the sample for freshness against each service's deployment or architecture history. - Look specifically at what is excluded and who decided: acquired estates, vendor-hosted components and internal tooling are where exclusions hide. ### Report it honestly The defensible form of the same claim states all three choices, for example: *94% of the 210 services in the primary organisation hold a model with threats enumerated and owned; against the full inventory of 340 services including the acquired estate, that is 58%; of 18 tier-1 services, 17 are covered and current, and the one gap is scheduled.* That sentence is longer and far more useful, and it survives the follow-up question that a single percentage does not. ### Why this matters beyond honesty Coverage drives decisions: whether to hire facilitators, whether to declare a rollout finished, whether an executive believes the design risk is understood. An inflated coverage number retires the programme's attention from exactly the estate that never got looked at. The acquired subsidiary in the example is the classic shape - a whole set of systems that no one is measuring, holding the same class of asset as the systems everyone is measuring.