Should a security programme mandate a CAPEC-derived attack-library sweep on every design review?
answer
- Gate the outcome, not the activity
- Curate per architecture archetype
- Coverage percentage is a gameable metric
- Near-zero matches means bad fit, not safety
- Feed escaped findings back into the subset
basics
~10 sMandate the outcome, not the traversal. A curated, architecture-specific library is a good floor for new teams, but percent-of-catalog coverage is theatre: require threats that came from the design too.
solid answer
~50 sI would mandate that every design produces a reviewed threat list with owners and decisions, and offer a curated library as the means - not mandate the traversal. The case for it is real: a repeatable floor, cheap to teach, productive on day one, comparable across a portfolio. The case against is that a full catalog walk scales with the catalog rather than the design, anchors the room away from business-logic abuse, and yields a coverage percentage that can be maximised without finding anything. So I curate per architecture archetype, pruned to what prerequisites can actually match and refreshed from what escaped us in testing and incidents. I ask for provenance - how many threats came from the design rather than the list - and I write an exemption path for off-catalog systems, where a near-zero match count means the library does not fit, not that the system is safe.
go deeper
Understand that following a security checklist is a starting point, not proof of safety, and that a checklist written for web services may barely apply to the system you are working on.
Be ready to explain the practical costs of an exhaustive catalog walk: effort that scales with the list instead of the design, unread volume, and false confidence from entries marked not-applicable in bulk.
Show that you would curate the library to the architecture in front of you, sequence it after the design pass, and challenge a sweep result whose applicable-pattern count is suspiciously low.
Own the standard itself: what you gate on, which metric you publish, who funds curation, how escaped findings feed back, and what the exemption path is for systems the catalog was never written for.
## The decision, stated honestly The question is not whether attack libraries are useful - they are - but whether *mandating* one as a gate is the right programme design. Mandates change behaviour in ways the mandate did not intend, so the answer has to account for what people will do once compliance is measured. ## The case for mandating **It creates a floor.** Without a standard, threat-modeling quality tracks the strongest person in the room, and most rooms do not have that person. A curated list guarantees that the well-understood attacks against this kind of component get considered on every design, every time. **It is cheap to teach.** "Walk this list and check the prerequisites against your design" is learnable in an afternoon. "Reason from your assets to novel abuse" takes years. For a portfolio staffed mostly by engineers who are not security specialists, the library is the only method that scales at all. **It makes results comparable and auditable.** When every team works from the same curated subset, you can see which teams are doing the work and which are not, and a reviewer can check the output without re-deriving it. ## The case against **Effort scales with the catalog, not the design.** A full traversal costs the same on a trivial internal service as on a payments boundary, which is exactly backwards, and produces volume that nobody triages. **It anchors.** A room handed twenty concrete catalogued threats switches from imagining to triaging. The threats that consist of your own business rules combining badly - typically the expensive ones - never surface, because nobody was thinking in that register. **Coverage is a gameable metric.** "We covered 100% of the catalog" can be achieved by marking entries not-applicable at speed. It measures traversal, not thought, and it manufactures confidence in exactly the systems that most need doubt. **Catalog shape is not universal.** Public attack libraries reflect the systems their contributors modelled, which skews heavily toward internet-facing web and cloud architectures. That skew is invisible until you point the library at something else. ## The off-catalog case Consider a legacy green-screen order-entry system reached through a screen-scraping middleware layer, used from branch offices. Point a web-oriented list at it and almost nothing matches: there is no browser, no session cookie, no JSON parser, no public endpoint. A team that reports "we ran the sweep, three entries applied, all mitigated" has produced a document that is worse than nothing, because it looks like assurance. The real threat model is a person on the branch network with legitimate access - an insider or someone who has taken over their workstation - and what is at stake is money and order integrity. The interesting questions are whether the middleware collapses many end users into one shared back-end identity (destroying attribution), whether the terminal protocol is authenticated and integrity-protected on the branch network, whether order amendments leave an audit trail the operator cannot alter, and whether anyone would ever notice a small, patient, well-formed fraud. None of that is catalogued, because nobody catalogues this shape of system any more. The correct reading of a near-zero applicable-pattern count is **the library does not fit this system**, never **this system has few threats**. A mandate without an exemption path teaches teams the opposite. ## How I would actually write the standard 1. **Mandate the outcome.** Every significant design change produces a threat list with, for each threat, an owner, a decision (mitigate, accept, transfer, eliminate) and a link to where the work landed. That is the artifact under review. 2. **Offer the library as the supported means, curated per archetype.** Maintain small subsets - untrusted-input ingest, multi-tenant API, batch integration, operator console, legacy terminal - rather than one giant list. Curation is the work that makes the library worth mandating. 3. **Sequence it second.** The design and asset pass runs first while the room is open; the library sweep is the completeness check afterwards. 4. **Require provenance, not coverage.** Ask how many threats came from the design rather than the list. A review where every threat is catalogued is a signal to look harder, not a pass. 5. **Write the exemption path.** Systems whose archetype has no curated subset get a facilitated design-driven session instead, and a note explaining why the library did not apply. 6. **Close the loop.** Every threat that escaped - found in testing, in an audit, in a real incident - is triaged for whether it belongs in a curated subset. A library that never changes is a library nobody is learning from. ## What I would watch Useful signals: the share of threats sourced from the design rather than the catalog; whether escaped findings from testing and incidents were represented in the relevant subset beforehand; how much of the threat list actually reached a backlog with an owner. Useless signal: percentage of catalog entries traversed. If the programme's only number is the last one, the mandate has become paperwork, and paperwork is what you get when you gate on an activity instead of an outcome.
- A team reports that only three catalog entries applied to their legacy terminal-based order system. How do you read that?As evidence the library does not fit that architecture, not as evidence of low risk. Public attack libraries skew toward the internet-facing web and cloud systems their contributors modelled. I would route that system to a facilitated design-driven session focused on the insider position on the branch network - shared back-end identities destroying attribution, unauthenticated terminal traffic, and whether order amendments leave an audit trail nobody can quietly alter.
- What metric would you use instead of percentage of catalog covered?Threat provenance and outcome. What share of threats came from the design pass rather than the list; how many reached a backlog with a named owner and a decision; and, looking backwards, whether findings that escaped into testing or incidents were already represented in the curated subset. Those measure whether thinking happened and whether the programme is learning. Traversal percentage measures neither and is trivially gamed.
- Who maintains the curated subsets, and how do you stop them going stale?A small central security-engineering function owns them, but the input comes from the field: every escaped finding from a penetration test, an audit or an incident is triaged for whether it belongs in an archetype subset. Retire entries nothing has matched in a year, and version the subsets so a team can see what changed since their last review. Curation is ongoing work, and an unfunded library decays into a checklist.
saying these in an interview costs you the question
- Gates on activity completed rather than on the threat list produced
- Reports percentage of catalog covered as an assurance metric
- Applies one giant catalog uniformly across every architecture
- Reads few applicable entries as evidence of low risk
- Never updates the library from escaped findings
- Leaves no exemption path for off-catalog system shapes