Outcome Metrics

## The four cells, and the one with no records behind it Detection outcomes are usually drawn as a confusion matrix: an alert fired and the activity was malicious (true positive), an alert fired and it was not (false positive), no alert fired and nothing malicious happened (true negative), no alert fired and something malicious did (false negative). Two of those four cells leave paperwork behind, and two do not. When a rule fires, a record is created. It has a timestamp, the fields that matched, an owner and — once someone works it — a disposition. That is why a **false-positive rate** is a real, computable number: the denominator is alerts raised in the period, the numerator is those closed as not-malicious, and both are columns in the case system. You can slice it per rule, per source, per shift. A **false negative** creates nothing. No rule matched, so no alert exists, so no analyst produced a verdict, so no row exists to count. The denominator is worse still: it is the set of intrusions that occurred, which is exactly the thing you were trying to measure. Nobody in the estate holds that set. The number is not merely hard to compute; there is no observation to compute it from. ## Silence is ambiguous, and that is the whole problem A month with no alerts on a detection has at least four explanations that look identical from inside the console: - the estate genuinely was quiet; - the log source stopped delivering — an agent died, a forwarder filled its disk, a cloud audit trail was disabled; - the rule stopped matching because a field was renamed or a vendor changed a schema; - an intruder was present and did nothing the rule describes. The first is good news, the last is the worst news you can have, and your own data cannot separate them. This is why mature teams instrument the *pipeline* separately from the detections: heartbeat checks that each source is still delivering, and volume baselines per source, so that at minimum silence can be attributed to a live feed rather than a dead one. Proving the feed is alive does not turn silence into evidence of safety — it only removes one of the four explanations. ## So how are misses counted at all? They are counted as **events, not as a rate**. A miss enters your records only when something outside the alerting pipeline surfaces it: an outside party tells you, an authorised exercise executes a behaviour and nothing fires, a hunt turns up activity with no alert behind it, or the reconstruction of a later intrusion walks back into an earlier one nobody caught. Each of those is a biased sampler — it finds a particular kind of intrusion and is blind to the rest — so the misses you know about are not a random draw from the misses that exist, and cannot be scaled up into a rate. The practical output is therefore a list, not a percentage: known misses, with the date, the behaviour, and the channel that revealed each one. That list supports two defensible statements — a floor (`at least this many got through`) and a shape (`every one of them was reported to us by someone else`) — and it supports no third statement about the total. ## Terms worth keeping straight - A **false positive** is a detection that fired on activity the rule does not correctly describe: the logic was wrong about what it saw. - A **benign true positive** is a detection that correctly identified the behaviour it names, performed by an authorised party — an administrator, a vulnerability scanner, a red team, a backup job. The rule was right; the activity was not hostile. Recording these as false positives makes a good rule look broken and hides the real tuning question, which is how to except the authorised source without excepting the behaviour. - A **false negative** is activity the detection set does not cover at all — no rule matched. It is a property of the whole estate and cannot be attributed to one rule. ## Why interviewers ask this It is a self-honesty probe. A candidate who says `our false-negative rate was under 2%` has either invented a denominator or does not know one is required. The expected answer names the missing denominator, explains that silence has several explanations, and then describes what you do instead: count known misses with provenance, measure whether your telemetry is arriving, and buy a denominator deliberately — by injecting behaviours you choose — while being clear that this measures your coverage of the chosen behaviours and not the behaviours real intruders will use.