Your threat model flagged an unauthenticated firmware-push path; a pentest did not exploit it. Is the finding closed?

### Two instruments, two kinds of evidence Threat modeling and penetration testing are often spoken of as interchangeable security activities, and the confusion surfaces the moment they disagree. They do not measure the same thing. A **threat model** is a design-time argument. It says: given this architecture, an actor in this position can do this to this asset, because nothing in the design prevents it. Its claims are about structure, and they hold for every build produced from that structure. A **penetration test** is an empirical result about a specific target. It says: within an agreed scope, over an agreed window, one or more testers with particular skills and particular starting positions attempted to compromise this deployed build and achieved these things. Its claims are about one instance, at one moment. The asymmetry matters. A successful exploit is strong evidence — it proves the path exists. A failed attempt is weak evidence, because the space of reasons for failure is enormous and mostly has nothing to do with the design being sound. ### Why the null result carries so little weight Consider the warehouse robot fleet controller. The model states that the firmware-push endpoint accepts an image without authenticating the pusher, so anyone who reaches the plant network can change what the robots do — an availability and physical-safety problem, not a data-theft one. Now the test comes back clean. Ask what the clean result actually rules out: - **Scope.** Was the tester ever placed on the plant network, or did they work from the corporate network with a firewall in between? Testers usually get the position the contract gave them, not the position the threat assumes. - **Environment.** Was there a real fleet controller with real robots to push to, or a staging instance with the push path disabled because nobody wanted to brick a lab unit? - **Time.** Testing is measured in days. Attackers are not. - **Skill and focus.** A generalist test covering an entire application will not spend its budget on one embedded protocol. - **Timing.** The build tested may differ from the design being modelled, in either direction. None of these are criticisms of testing. They are the ordinary properties of a time-boxed, scope-limited engagement, and they are exactly why absence of a successful exploit cannot be read as evidence of design soundness. ### What actually closes a design-level finding Three things legitimately end it: 1. **The design changes** so the threat no longer applies — the push path now requires an authenticated, authorised operator, or the controller only accepts images bearing a signature it verifies against a key it holds. 2. **A control is shown to stand in the path and work.** Note the second half: asserting that a control exists is not the same as showing it operates. This is one of the places a test genuinely helps. 3. **Someone with the authority accepts it**, in writing, with the reasoning recorded. That is a decision, not a null result, and it survives review because a person's name is on it. What does not close it: a clean report, a tester's opinion, or the passage of time. ### Where a test does help a model The productive relationship runs the other way round. A model is full of assumptions — "the controller is only reachable from the plant VLAN", "the loader rejects unsigned images", "that admin interface is not exposed". Those assumptions are precisely what a test is good at checking, and turning them into explicit test targets is a much better use of the engagement than asking a tester to re-derive the threat list from scratch. Model first, then aim the test at the assumptions the model leans on hardest. The reverse also carries signal: a tester finding something the model never mentioned tells you the model's scope or its picture of the system was wrong, which is worth more than the individual finding. ### The organisational failure mode Watch for the sequence where a design finding is uncomfortable, a test is scheduled, the test comes back clean, and the finding quietly disappears. It is rarely dishonest — it feels like evidence-based decision-making. It is not, and the tell is that the design has not changed. If the architecture still permits the thing, the threat is still there; all that has changed is that one attempt failed. Say so plainly, in those terms, and re-anchor the conversation on which of the three legitimate closures the team wants to pick.