skip to content

400 of your SOC's 900 detection rules have never fired - what does that entitle you to conclude?

level: seniorimportance: should knowfreq 44%

answer

  1. one observation, several causes
  2. sort the tail before opening it
  3. did it execute, or did it match
  4. nothing looks exactly like nothing
  5. recall is not computable from output

basics

~20 s

A never-fired rule has at least four causes - never really enabled, no telemetry reaching it, the behaviour genuinely absent here, or logic that cannot match - so portfolio silence proves nothing about how much you are missing.

solid answer

~50 s

At portfolio scale I partition the tail rather than investigate 400 rules one at a time. Cheap discriminators sort most of them without opening a single rule: was it enabled and executing all window; does its log source exist here; does it reference fields the parser populates; do we even run the platform it targets. That splits the 400 into deployment facts, telemetry gaps, honest zeroes for behaviour that does not occur, and a residue of rules that should have matched and did not - and only that last group is worth per-rule diagnosis. The inference limit matters more than the partition: a detection's absence of output is indistinguishable from a quiet estate, so you cannot compute a false-negative rate from the portfolio's own data. The only evidence a silent rule still works is executing the behaviour it claims to catch and watching for the alert.

go deeper

for a junior

Know that a rule producing no alerts is not automatically a working rule or a broken one. The same silence comes from a disabled rule, a missing log source, and an estate where the behaviour simply does not happen.

for a middle

Be able to name the cheap checks that sort a large silent tail without opening each rule: did it execute, does its log source exist here, does it reference fields the parser populates, does the estate contain those assets.

for a senior

Demonstrate the inference limit under questioning. Explain why recall is not computable from a portfolio's own output and why executing the behaviour is the only evidence a quiet detection still functions.

for a principal

Decide what the coverage number you publish actually asserts. A rule count read as a protection count is a claim your organisation may act on, and owning the partition behind it is what makes the claim defensible.

## Silence is a measurement with several causes 'Never fired' is one observation with at least five explanations, and they demand different responses. At 400 rules you cannot open them individually, so the job is to sort the tail into buckets using data you already have. **1. It was never really running.** The rule exists in the console but is disabled, scheduled and erroring, saved as a draft, or deployed to one tenant and not the others. This is a deployment fact, not a detection fact, and it is usually the largest bucket in a portfolio nobody has audited. The discriminator is the rule's own execution metadata: did it execute, not did it match. **2. The telemetry never arrives.** The rule queries a source or an index this estate does not onboard, or references a field the parser here does not populate. A rule watching for a command line on a platform where command-line capture was never enabled will execute happily forever and match nothing. The discriminator is a schema check: does every field this rule filters on appear in real records from this estate in the window? **3. The behaviour genuinely does not occur.** A rule for a cloud service you do not use, a database engine you retired, an operating system nobody runs. This is an **honest zero** - the rule is fine and there is nothing to see. The discriminator is asset inventory, not the rule. **4. The behaviour occurs and the logic cannot match it.** Over-constrained conditions, a field renamed by a platform upgrade, a value the vendor now writes in a different case. This is the dangerous bucket because it is the one that looks identical to bucket three from outside. Telling one broken rule from a genuinely quiet estate is a per-rule investigation and it is a different job from this one - what portfolio partitioning does is shrink the candidate list from 400 to something a person can actually work. **5. It fired but never opened a case.** If you counted in cases rather than firings, rules that only ever corroborate somebody else's alert appear in the never-fired column. That is a counting artefact, and it is why the last-fired timestamp from execution metadata belongs in the table beside case counts. ## The inference limit, which is the real answer Whatever the partition says, the tail cannot be converted into a statement about your exposure. A detection produces output when it matches and produces nothing otherwise, and *nothing* is exactly what a quiet estate produces too. The two are indistinguishable in the data. That has a hard consequence: **the false-negative rate of a detection portfolio is not computable from the portfolio's own output.** You can measure precision over things that fired. You cannot measure recall over things that did not. This is what separates a detection from most other rules an organisation runs. A gate that blocks changes can be replayed against captured inputs and you can watch what it decides. A detection's input is the adversary's behaviour, and replaying a stored record only proves the parser and the query still work on that record. **The only evidence that a silent detection still works is executing the behaviour it claims to catch in the estate and confirming the alert appears** - emulation, a purple-team exercise, an authorised test of the technique against a real host with real telemetry. Absent that, a green rule in a console is an assertion, not a measurement. ## The reporting risk A leader shown '900 detections' reads 900 protections. The tail is where the gap between that impression and reality lives, so a content review that reports rule counts without reporting the never-fired share is producing a misleading number even if every individual figure in it is accurate. Reporting the partition - deployed but silent, no telemetry, no such assets, unexplained - converts an embarrassing statistic into an actionable one. ## What this does not decide Partitioning the tail tells you which rules are silent and roughly why. It does not tell you which to remove, and the temptation to convert a long tail straight into a deletion list is exactly where portfolios go blind: every removal is a decision about what you accept not seeing, and that decision has to be priced separately from the measurement that surfaced it.

  • Why can you measure a detection's precision but not its recall?
    Precision is computed over things that fired: you have the alerts, the cases and the verdicts, so the share that was right is a countable quantity. Recall needs the denominator of everything that should have fired, and the only record of that is the adversary's behaviour, which you did not observe - if you had, the detection would have caught it. The missing denominator is missing in principle, not just in practice.
  • A vendor says their rule pack is validated. Does that transfer to your estate?
    No. Validation in the vendor's lab proves the logic matches records shaped the way they shaped them. Whether your parsers populate the same fields, your agents capture the same detail, and your platform version writes the same values is a property of your estate. The claim only transfers if the behaviour is executed here and the alert is observed here.
  • How do you report a 44 percent never-fired share without it sounding like negligence?
    Report the partition rather than the raw share. 'Of 400 silent rules, 150 target platforms we do not run, 120 query telemetry we never onboarded, 90 were never enabled, and 40 are unexplained and under review' is four different problems with four different owners. The raw percentage invites a mass cleanup; the partition invites the right four decisions.

Four hundred smoke detectors that have never sounded is not evidence of four hundred fireless rooms. Some have no battery, some are in rooms with no door, and you only find out which is which by lighting something.

saying these in an interview costs you the question

  • Reads a never-fired rule as proof nothing happened
  • Treats the never-fired share as the SOC's false-negative rate
  • Assumes silence always means the rule is broken
  • Proposes deleting the tail straight off the report
  • Claims replaying stored logs proves a detection still works

context