Why does a SOC sample already-closed alerts and re-review their verdicts?

## The problem this practice exists to solve Every alert a SOC works ends in a disposition, and the overwhelming majority end in some form of "not an incident". That verdict is then treated as fact by everything downstream: the queue empties, the metrics count it, the tuning backlog reads it, and the case is never opened again. What makes that dangerous is the **asymmetry of the two error types**. If an analyst wrongly escalates a harmless case, the error surfaces almost immediately - a second analyst, an incident lead or an engineer whose host is about to be isolated pushes back, and the mistake is corrected within hours. If an analyst wrongly closes a real intrusion, **nothing happens at all**. There is no failing test, no bounced ticket, no user complaint. The adversary is actively working to keep it that way. The error is silent by construction, and it stays silent until the intrusion becomes loud enough to be found some other way - often by a third party telling you. So a closed verdict is the only major SOC output with no feedback path. A verdict audit builds one on purpose. ## What the practice actually is The core loop is a **blind re-review** of a sample: 1. **Draw a sample** of closed cases from a period recent enough that the underlying telemetry still exists. 2. **Strip the answer.** The reviewer gets the original alert and the raw evidence, not the disposition, the case notes or the name of the analyst who closed it. 3. **Re-work the case independently** to a fresh verdict. 4. **Compare** the two verdicts, record disagreements *before* anyone discusses them, and adjudicate the disputes with a third, more senior reviewer. 5. **Convert findings into content changes** - a rule that needs context, an enrichment step that was missing, a playbook that told the analyst to close on a weak signal, a piece of authorised-activity reference data nobody had. The important design commitment is in step 5. A verdict audit is **content quality control**, not an examination of people. The moment its output is a per-analyst overturn table, analysts stop closing anything and the queue - and the data - collapse. ## What the audit can and cannot tell you It can tell you that a class of cases is being systematically mislabelled: for example that alerts on an administrator's credential-audit tooling are routinely recorded as "false positive" when the rule was in fact correct and the activity merely authorised. That is a finding with real teeth, because a false-positive label is the normal justification for suppressing a rule, and suppressing it carves out exactly the behaviour an intruder would reuse. It cannot certify the population. Thirty re-reviewed cases out of forty thousand closures bound the gross error rate very loosely, and a targeted intrusion is a rare event that a small random sample will usually miss entirely. "We sampled thirty and found nothing" means the process is not obviously broken; it does not mean the queue is clean. It is also bounded by **retention**. You can only re-review a case while the evidence behind it still exists - the process telemetry, the sign-in logs, the proxy or flow records the original analyst looked at. Once those age out, the case is un-auditable, which is why the audit runs on a cadence inside the hot retention window rather than as an annual exercise. ## How it differs from neighbouring measurements Counting how many alerts a rule produced tells you about the rule's output volume. Counting how many were escalated tells you what analysts did with it. Neither tells you whether the *verdicts* were right, because both are computed from the same dispositions whose correctness is in question. That circularity is the reason the audit has to go back to the raw evidence rather than to the ticket text. ## What good looks like A mature programme runs a small blind sample on a regular cadence, seeds it with a few cases whose true answer is already known (an in-scope red-team action, an alert that preceded a confirmed incident), reports findings at the level of rules and playbooks, tracks each finding to a named owner and a fix, and publishes what changed. Analysts cooperate with an audit that repairs their tooling; they defend themselves against one that grades them.