How do you handle the steps of an abuse case that no automated test can cover?
answer
- decompose before you give up
- social, physical, collusion, third party
- silent omission is the real defect
- downshift to the nearest testable invariant
- reason, substitute exercise, cadence
basics
~10 sSplit the narrative into steps, automate the technical invariants, and keep each human-dependent step visible in the model with a stated reason and a named substitute exercise instead of quietly deleting it.
solid answer
~50 sSome steps are not scriptable: social engineering, physical access, collusion between two people, anything needing a third party's cooperation. The wrong move is silent omission — the tester cannot automate it, so it disappears and the model looks fully covered. The right move has three parts. First, decompose: a field-service abuse case reading "befriend the cleaning contractor, borrow a badge, reach the depot terminal" has one social step and several technical ones. Second, downshift to the nearest testable invariant: you cannot script the befriending, but you can assert that a badge deactivated in the personnel system is refused by the terminal within the stated window. Third, record the residual step in the model itself, with the reason it resists automation and the substitute — a tabletop walkthrough or a physical red-team exercise on a stated cadence. Then say plainly that a green suite covers the automated steps, not the abuse case.
go deeper
Know that some attack steps involve people or physical access and cannot be turned into a test, and that the right response is to flag them rather than quietly leave them out of the plan.
Be able to break an abuse-case narrative into steps and say which are automatable, and to propose the technical assertion that sits immediately after a social or physical step.
Show you record the residual properly — reason, substitute exercise, cadence — and that you look for the chokepoint after the untestable step instead of abandoning the abuse case.
Own the coverage claim itself: define what a green suite is permitted to mean, report covered versus exercised versus accepted separately, and push for design changes that eliminate steps nobody can verify.
## Why this comes up at lead level Once a team starts converting abuse cases into negative tests, a quiet distortion sets in: the abuse cases that translate cleanly get tests, and the ones that do not get dropped. Nobody decides this; it is the path of least resistance for whoever is writing the suite that week. The result is a coverage story that is honest about the easy half of the threat model and silent about the hard half — and the hard half is often where the highest-impact scenarios live, because insider and physical scenarios rarely reduce to an HTTP request. ## What actually resists automation - **Social steps.** Persuading, phishing, pretexting, impersonating over the phone. There is no assertion for whether a human will be convinced. - **Physical steps.** Tailgating, borrowing a badge, reaching a terminal in a locked facility, plugging something in. - **Multi-party collusion.** Two authorised people combining their legitimate access to defeat a separation-of-duties control. Each half is permitted, so no single call is refusable. - **Steps needing a third party.** Anything that depends on a vendor, a partner, or a supplier acting — you cannot make their side happen on demand in a test run. - **Judgment steps.** A reviewer approving something they should have questioned, an operator overriding an alert. ## The move that separates strong candidates: downshift An un-automatable step almost always sits next to a technical invariant that *is* testable. The interesting question is not "can I automate this step" but "if this step succeeded, what must still stop the attacker, and can I assert that?" Take a field-service app whose abuse case reads: *befriend the cleaning contractor, borrow a badge, reach the depot terminal, and pull the route and customer data*. The befriending is untestable. Almost everything after it is not: | Step | Handling | | --- | --- | | Persuade the contractor to hand over a badge | Not automatable — substitute exercise, reason recorded | | Badge opens the depot door | Not automatable in software — physical control, reviewed | | Badge revoked in the personnel system still works | Automatable — assert refusal within the stated window | | Terminal session established with badge alone | Automatable — assert a second factor bound to an individual is required | | Bulk export of customer routes from that session | Automatable — assert the export limit and that the attempt is recorded | The abuse case survives as a whole even though one step never becomes a test, and the tests that do exist are the ones that would actually stop the attack at its later stages. ## Keeping the residual visible For every step you cannot automate, record three things in the model beside the step: **why** it resists automation, **what verifies it instead** — a tabletop walkthrough of the narrative, a scheduled physical or red-team exercise, a procedural control whose own evidence you can inspect — and **how often** that substitute happens. "Manual" written alone is the failure mode: it reads as covered while committing nobody to anything and producing no evidence. This is also where you resist a tempting fake. Scripting an email to see whether a contractor replies is not a test of the abuse case; it produces a number about one moment and no invariant, and it has consequences for real people that a test suite should not be creating on its own. ## The claim you are allowed to make The organisational half of the answer is what a lead owns: stating precisely what the green suite means. It means every abuse-case step that reduced to an automatable invariant currently holds. It does not mean the model is closed. Reporting coverage as a split — steps covered by standing tests, steps covered by a named recurring exercise, steps accepted with a written reason — keeps the conversation honest and makes the un-automatable set visible enough that someone can argue for a design change that removes it. That last outcome is the best one available: the strongest response to a step you cannot test is often to redesign so the step no longer leads anywhere.
- Give a concrete example of downshifting a social step to a testable invariant.The abuse case starts with persuading a contractor to lend a badge. That step stays manual. The next steps do not: assert that a badge revoked in the personnel system is refused by the depot terminal inside the stated window, that establishing a session needs a second factor bound to a named individual, and that a bulk export from that session is capped and recorded. The attack has three later chokepoints and all three are testable.
- What is wrong with simply labelling the step "manual test" and moving on?It reads as covered while committing nobody to anything. There is no cadence, no named exercise, no evidence produced, and no reason recorded, so a year later nobody can say whether it was ever verified. Write the reason it resists automation, the substitute exercise, and how often that exercise runs — otherwise "manual" is indistinguishable from "never checked".
- How do you report coverage to a team that reads a fully green suite as a closed threat model?Split the claim. Say how many abuse-case steps have standing automated assertions, how many are verified by a named recurring exercise, and how many are accepted with a written reason. A single percentage invites the wrong conclusion. The split also makes the un-automatable set visible enough that someone may propose a design change that removes the step entirely, which is the best outcome available.
You cannot test whether a stranger can talk their way past a receptionist, but you can test that the lift refuses a badge that was cancelled this morning.
saying these in an interview costs you the question
- Drops the step because it cannot be scripted
- Claims the threat model is closed because the suite is green
- Proposes scripting a phishing attempt as the automated test
- Writes "manual" with no cadence, evidence or reason
- Never looks for the testable invariant after the social step
- Assumes collusion cases are untestable end to end