How do you use incident data to show a green compliance control is not reducing risk?
answer
- bring a number the pass rate cannot answer
- incidents mapped to control status
- prevention leaves no incident
- mechanism plus data, not data alone
- arrive with the replacement already written
basics
~20 sMap each of the last dozen incidents to the controls that were green throughout it. That table turns opinion into measurement, and the argument it supports is a change to the control's wording, brought with a replacement check.
solid answer
~50 sTake the last twelve incidents and near misses, and for each one record which mapped controls were passing at the time. When every credential incident happened while rotation and quarterly access review were both green, and none of them involved a password older than 90 days, you have a measurement rather than an opinion. Then be honest about what it does not prove: prevented incidents leave no trace, so absence of incidents is never evidence a control works, twelve is a small sample, and you only see what you can detect. So pair it with leading indicators that have variance — time from a leaver event to access revocation, the proportion of privileged accounts with a phishing-resistant factor. Finally, arrive with the replacement control already written and checkable, so the person measured on pass rate is being offered a swap, not a hole in their report.
go deeper
Know that a control being green and an incident happening are compatible, and that comparing the two is how the case gets made rather than by asserting a control is pointless.
Be able to build the mapping: incidents on one axis, the control status recorded at the time on the other, sourced from stored evidence rather than memory.
Demonstrate the limits unprompted — survivorship, small samples, detection bias — and pair the incident table with a leading indicator that has variance.
Own the incentive problem: the person you are persuading is measured on pass rate, so a swap with parallel running is the only proposal that can be accepted.
## Why measurement and not argument In a control-review meeting, the engineer says the rotation control is pointless and the compliance backlog owner says it is a required control that is 100 percent green. Both statements are true and the conversation goes nowhere, because one side is arguing about effectiveness and the other about satisfaction. The way out is to bring a number that the pass rate cannot answer. ## Building the table Take the last twelve incidents — and near misses, because there are more of them and they are less politically loaded — and for each one record three columns: | Incident | Controls mapped to this risk | Control status throughout | |---|---|---| | Credential reuse on an admin console | rotation, quarterly access review | both green | | Leaver retained repo access 6 weeks | quarterly access review | green | | Session token replayed after reset | rotation | green | Define 'green throughout' precisely, because it is the column that will be challenged: the control was passing in the reporting period containing the incident, on the specific systems involved. Pull it from the stored evidence rather than reconstructing it, and be ready to show the run record for each row. The finding you are usually after is not 'the control failed'. It is stronger and stranger than that: **the control never failed, and the incidents happened anyway, by paths the control does not describe**. Add a fourth column if you can — what would have had to be different to prevent this — and the replacement control usually writes itself. ## What this proves, and four ways it can mislead Be the person in the room who states the limits before someone else does. **Survivorship.** Prevented incidents leave no incident. A control that works generates an absence, and absence looks identical to irrelevance in this table. This is why the argument must be about a **mechanism**: rotation cannot prevent phishing, not merely 'no phishing incident was prevented by rotation'. Mechanism plus data is an argument; data alone is not. **Small numbers.** Twelve incidents is a handful of events over an uneven period. Treat the table as an existence proof of the failure mode, not as a rate. **Detection bias.** You only enumerate incidents you detected. If your detection is weighted toward one class, so is your table, and the control you are attacking may look worse than it is. **Obligation is orthogonal.** A control can be contractually or legally required with zero incident linkage. Demonstrating ineffectiveness does not by itself authorise removal, and pretending otherwise loses the room. ## Better instruments than incident counts Incidents are rare, lagging and noisy. The stronger measurements are leading indicators with **variance** — numbers that move when the exposure moves: - Time from a leaver event in the HR system to last access revoked, reported as a distribution with its tail, not a mean. The mean hides the one account that lived for six weeks. - Proportion of privileged accounts with a phishing-resistant factor enrolled, trended. - Count of accounts active with no sign-in in 90 days. A control whose metric has been 100 percent for eight quarters is measuring something that cannot get worse — which usually means it is not measuring exposure. Metrics with variance are the ones worth gating on, and offering one is what makes you credible rather than merely negative. ## Making the case to someone measured on pass rate The compliance backlog owner is not the obstacle; their incentive is. If your proposal reduces the number of green controls, you are asking them to look worse in exchange for an argument about mechanism. So do three things: 1. **Attack the definition, never the person or the function.** The sentence is 'this control is satisfied and its objective is not met', which concedes their entire position and moves the discussion to wording. 2. **Bring the replacement, already checkable.** Name the new control, the query behind it, the system of record it reads, who owns it, and what it will report on day one — including that it will start red, and by how much. A swap keeps the report intact; a deletion does not. 3. **Run both for a reporting period.** Keep the old control green while the new one bakes. That removes the coverage-gap objection and gives you a real first data point before anything is retired. ## The outcome to expect Often nothing about any system changes on the day you win. The estate is unchanged; what changes is the **control's definition** — the framework objective is re-implemented as a check on the property that actually matters, and both the old and new evidence exist for the transition period. That is the whole point of this exercise: the artefact you are editing is the wording of the control, and the data is what earns you the right to edit it.
- The control owner says your twelve incidents prove nothing because prevented ones are invisible. How do you answer?Concede it immediately, because it is correct, then move to mechanism. Rotation cannot prevent a phished credential used within minutes, or a replayed session token — not empirically, structurally. The table shows the failure mode is real and recurring; the mechanism shows the control could not have prevented it. Data alone is rebuttable by survivorship; mechanism plus data is not.
- What single metric would you add to the report alongside pass rate?Time from a leaver event to last access revoked, reported as a distribution with the 95th percentile, not a mean. It has variance, it degrades visibly when deprovisioning breaks, it maps to a control objective auditors recognise, and it is queryable from the HR system and the identity provider. Averages hide exactly the account that mattered, so publish the tail.
- How long do you run the old and new controls in parallel?At least one full reporting period, so the new control has been evidenced end to end before anything is retired, and the transition never shows a coverage gap. Use that period to see the new control's real red rate and to fix its false positives while nothing depends on it. Retire the old control at the period boundary, with both evidence sets on file.
saying these in an interview costs you the question
- Presents incident counts as proof without the mechanism
- Ignores that prevented incidents leave no evidence
- Proposes deleting a control with no replacement
- Treats the compliance owner as the adversary
- Uses a mean instead of the tail for time-to-revoke