Automation filed 180 generated threats as tickets overnight and the team closed most in a day. What does that tell you?
answer
- bulk closure is a price signal
- raw material versus a decision
- the durable cost is trust in the channel
- generate, then curate, then file
- file deltas, not periodic dumps
basics
~20 sIt tells you auto-filing moved review work onto a sprint board without doing any of it. Mass closure is the team correctly rejecting undifferentiated output, and the real cost is that the next genuine finding gets closed the same way.
solid answer
~50 sTwo things. First, the output was undifferentiated: 180 items arriving at once, none carrying context about this system's assets or assumptions, is a dump rather than a set of decisions, and closing them in bulk was a rational response. Second, and worse, the team has now learned that tickets from this source are noise — so the one genuinely serious threat in that batch was closed with the rest, and the next batch will be too. I would turn auto-filing off as a default and make the flow generate-then-curate: the machine produces the candidate list, a human pass merges duplicates, discards what the design already handles, adds business impact, and only what survives becomes a ticket with a named owner. Filing volume is not a feature; every filed item should be something a person decided was worth someone's week.
go deeper
Know that a generated threat is a candidate, not a task. Someone has to decide it matters before it becomes work for an engineer.
Explain why a large undifferentiated batch gets mass-closed: no impact, no owner, heavy duplication. Be able to describe a curation pass that turns a long list into a few defensible items.
Show that you would change the pipeline rather than the team's behaviour — curate before filing, send change-only deltas, and reserve auto-filing for machine-decidable model defects.
Own the trust cost and the metrics. Be ready to argue that you are accountable for the tickets you create, and to replace a threats-identified metric with mitigated-or-accepted and assumptions-verified.
## Read the signal, not the number The interesting fact is not that a tool produced 180 threats — a per-element rule set over a decent-sized model will do that easily and is arguably working as designed. The interesting fact is the team's response, because it is a measurement of how much decision-making was embedded in the output. **Bulk closure is the market price the team put on those items, and it was near zero.** A threat that reaches an engineer as a ticket is asking for a slice of a sprint. To be worth that, it must arrive with things the generator does not have: which asset it puts at risk, what the design already does about it, whether it is a duplicate of the other eleven instances of the same rule, and who should own it. Strip those out and you have shipped raw material into a queue meant for decisions. ## The two costs **The visible cost** is the day of triage, which is real but recoverable. **The durable cost** is trust. Once a source of tickets is known to be noise, it is filtered by habit — and filtering is indiscriminate. Somewhere in the batch was the threat that mattered, and it was closed in the same sweep with the same comment. You have not just wasted a day; you have degraded the channel through which future security work has to travel. Repeat the run next month and the batch will be closed faster, not slower. There is a third effect worth naming in a senior conversation: metrics distortion. A programme that reports 'threats identified' will look excellent, while 'threats mitigated' quietly stays flat, and the gap between them is invisible unless someone asks. Volume is the easiest security metric to manufacture and the least informative. ## What good looks like **Generate, then curate, then file.** The generated list is an input to a review, not an output of one. A human pass over it should: - **Collapse duplicates.** Eleven instances of the same rule across eleven flows are usually one design decision, not eleven tickets. - **Discard what the design already answers.** If the platform handles a category by construction, the rule firing is not news. - **Add impact.** Attach what each surviving threat means for this business and its assets — the piece no rule set carries. - **Assign an owner.** An item with no named owner is a backlog fossil in waiting. Only what survives becomes a ticket. In practice the 180 becomes something in the range of a handful to a dozen, each defensible in a sentence. **Let automation file the structural failures instead.** There is a narrow class where auto-filing is genuinely fine: unambiguous, low-volume, machine-decidable defects in the model itself — an element with no trust boundary, a model that has not been reviewed since the design changed. These are true by construction, so a person is never asked to adjudicate taste, and the volume is naturally small. **Keep the diff, drop the dump.** When a model is re-run, what deserves attention is what *changed*: new elements, threats that went live because an assumption flipped, items previously accepted that now apply differently. A steady dribble of deltas is consumable in a way a periodic flood is not. ## The organisational call As the person who owns the programme, the decision to defend is that **you are accountable for the number of tickets you create, not just their quality**. Every filed item spends someone else's attention, and attention spent on noise is subtracted from the work you actually want done. A useful internal rule: no automated pipeline may open a ticket that no human has looked at, with the exception of the small machine-decidable class above. If leadership wants a volume metric, offer threats mitigated and assumptions verified instead — both are hard to fake and both track the thing the programme exists to move. ## Interview framing Avoid two failure modes when answering. Do not treat the team as lazy — their response was correct given what they received. And do not declare tooling useless — the generation step did fine work; the defect was in wiring its raw output directly to a queue that expects decisions. The candidate an interviewer wants is the one who says: the machine's job ended at the candidate list, and somebody skipped the step between that and the sprint board.
- Is there any generated output you would let file a ticket automatically?Yes, a narrow class: machine-decidable defects in the model itself. An element or store with no trust boundary, a model whose design has changed since it was last reviewed, a required field left empty. These are true by construction, so nobody is asked to adjudicate a judgment call, and the volume is small enough that the tickets stay credible. Anything requiring an impact judgment goes through a person first.
- How would you measure whether the threat-modeling programme is working, if not by threats identified?By what moved. Threats mitigated or explicitly accepted with a named owner, load-bearing assumptions verified, and the share of designs that were modelled before build rather than after. Identified counts are trivially inflated by pointing a generator at more models, so they reward volume rather than outcomes and tell you nothing about whether anything got safer.
- The team asks you to stop running the generator entirely. How do you respond?Keep the generator, change what leaves it. The generation step gives cheap coverage and catches structural gaps, which is worth having; what failed was piping raw output into their queue. I would commit to filing only curated items with impact and an owner, switch periodic dumps to change-only deltas, and let them measure me on how many filed items they judge worth doing.
saying these in an interview costs you the question
- Blames the team for not triaging 180 items properly
- Counts threats identified as programme success
- Treats auto-filing volume as evidence of coverage
- Concludes that threat-modeling tools are useless
- Files every generated item to be thorough