How do you compute per-rule true-positive yield across 900 SIEM detection rules?

## Two datasets, one join Per-rule yield is not a metric your platform ships; it is a join you build. **Side one - rule-execution metadata.** Every detection platform records, per rule: a rule id, an enabled/disabled state, a schedule or streaming mode, when it was deployed or last edited, how many times it executed, how many times it matched, and when it last matched. This is the side that tells you a rule exists and whether it produces anything. **Side two - case dispositions.** The case management system records, per case: which alert opened it, which rule that alert came from, who worked it, how long it took, and the closing verdict. This is the side that tells you whether the firings meant anything. The join key must be a **stable rule id**, not the rule's name. Names get edited; a portfolio that joins on display name silently splits one rule's history into three the moment somebody fixes a typo. ## Fix the unit before you compute anything The single largest error in this measurement is mixing units. Three candidates, and they differ by orders of magnitude: - **firings** - every match. One noisy rule can match half a million times on a fleet. - **alerts reaching a human** - firings that survived aggregation and grouping. - **cases** - what an analyst actually opened a verdict on. One case can swallow hundreds of firings from the same rule. Yield expressed as *cases per firing* and yield expressed as *escalations per case* are both defensible; a table that quietly uses one for some rules and the other for others is not. Write the definition down and hold it across the whole portfolio, because the entire point of this exercise is comparing rules to each other. ## Normalise, or the ranking is an artefact of age A rule live for four years and a rule deployed six weeks ago cannot be compared on raw counts. The minimum normalisation is **per rule-day live** - firings and cases divided by the days the rule was enabled during the window. Two further denominators matter in a real estate: the **number of assets or accounts in the rule's scope** (a rule scoped to twelve domain controllers will never out-fire one scoped to forty thousand workstations), and **whether the rule was enabled for the whole window** at all. Rules disabled mid-window for an incident or a migration produce a truncated numerator against a full denominator, which makes healthy rules look dead. ## Attribution decides who gets the credit Most case systems attribute a case to the rule whose alert *opened* it. That choice has consequences: - A rule that fires ten minutes later and confirms the same activity records zero cases forever, even though it is doing real work. If your platform supports it, count **contributing** firings on a case separately from **originating** ones, and report both. - Rules that feed a risk-scoring or aggregation layer instead of raising their own alerts will show a firing count and no cases at all. They need their own bucket; scoring them as zero-yield detections is simply wrong. - Deduplication and suppression upstream of the queue mean some firings never reach a case by design. If suppression happens in the pipeline rather than the rule, your denominator includes matches that no human could ever have converted. ## The disposition vocabulary is the ceiling on quality Yield inherits every weakness of the closing verdicts it counts. Free-text closures are uncountable. A vocabulary with only *true positive* and *false positive* forces analysts to record authorised, correctly-detected activity as a false positive, which makes a rule that is working perfectly look defective. At minimum the closing codes need to separate: hostile activity confirmed; authorised activity correctly identified (benign true positive); the rule was wrong about the records; and insufficient evidence to decide. Whether those recorded verdicts are themselves *correct* is a different exercise from this one, and it is worth saying so out loud rather than pretending the join produces truth. ## What you produce One row per rule: id, name, days live, firings, firings per rule-day, cases originated, cases contributed to, escalations, verdict mix, last fired. Read as a distribution rather than a league table, that single table is what a content review actually argues over.