How do you set an anomaly-detection threshold that trades false alarms against detection delay for on-call?

## The tradeoff is a curve, and the curve is the deliverable Every detector has one knob that trades the same two quantities in opposite directions: - **False-alarm rate** — how often it cries wolf when nothing is wrong. - **Detection delay** — how long a genuine change runs before it is caught. Tighten the threshold and you find things faster and wake people for nothing more often. Loosen it and the pager goes quiet while incidents run longer. No amount of cleverness in the detector removes the tradeoff; a better detector moves the whole curve inward, but you still choose a point on it. The standard vocabulary is **average run length (ARL)**. The in-control ARL is the expected number of observations between false alarms while the process is fine — bigger is quieter. The out-of-control ARL is the expected number of observations to detect a shift of a given size — smaller is faster. Quoting both, for the specific shift size worth catching, is what separates an engineering answer from "we used three sigma". ## Convert statistics into operational units ARL in "observations" persuades nobody. Convert it. If the series is sampled every minute and the in-control ARL is 1,000 points, that is a false alarm roughly every 17 hours — about ten a week from **one** series. Detection delay of 10 points is 10 minutes of undetected incident. Now the conversation is about things people can price: an on-call engineer's tolerance is a handful of actionable pages per week, and the cost of an undetected outage is revenue or trust per minute. A rough decision frame: choose the threshold that minimises expected total cost, `alarms_per_week * cost_per_false_page + incidents_per_week * cost_per_minute * detection_delay`. The numbers will be soft, but writing them down forces the real conversation — how much is a 3am page worth versus five more minutes of downtime — and that conversation is the actual deliverable at a senior level. ## Fleet multiplication is where thresholds die The most common production failure is tuning per series and deploying across a fleet. A false-alarm rate of one per thousand points per series is fine alone. Run it on 500 series and the expected rate is `500/1000 = 0.5` alarms per time step — one every two points, forever. So set the budget globally first — "no more than N actionable pages per on-call week across all monitors" — and derive per-series thresholds from it. Practical levers: - **Tier the series.** A handful of top-line metrics get sensitive thresholds and page; the long tail gets loose thresholds and a dashboard. - **Aggregate before detecting.** Monitoring one well-chosen aggregate beats monitoring a thousand slices, each of which alarms independently. - **Correct for multiplicity.** If you insist on per-slice detection, the per-series threshold must tighten as the fleet grows, otherwise the global rate grows linearly with it. ## Levers other than the threshold The threshold is not the only knob, and the others often buy a better trade: - **Persistence (k-of-n).** Require, say, three consecutive breaches before alarming. Random breaches rarely repeat, so false alarms fall steeply while delay grows only by the extra points — usually the best value per unit of delay available. - **Accumulate rather than threshold per point.** Cumulative methods catch a small sustained shift that a per-point rule at the same false-alarm rate never sees, buying detection power at no false-alarm cost for the shift sizes they are tuned to. - **Severity tiers.** Page, ticket, dashboard. Most detections do not need a human awake; routing by severity lets you run a sensitive detector without a punishing page rate. - **Suppression windows.** Mute known deploys, migrations and scheduled events. An alarm you already know the cause of is pure noise and trains people to ignore the channel. - **Corroboration.** Require two independent signals to agree before paging. This cuts false alarms far more than it costs in delay, because independent false alarms rarely coincide. ## Measure the curve; do not assert it Backtest on labelled history: take incidents you know about with their start times, replay the detector across a grid of thresholds, and plot alarms-per-week against median detection delay. Present that curve to whoever owns the pager and let them choose the operating point — the choice is theirs, the measurement is yours. Two cautions. Labels are incomplete: you know the incidents that were noticed, which biases delay estimates optimistically for loud failures and hides the quiet ones entirely. And the curve drifts as the system changes, so it needs re-measuring, along with an ongoing count of what fraction of pages were actionable. A precision rate below about a third destroys trust in the channel, and once on-call stops reading the alerts your effective detection delay is infinite regardless of what the threshold says. ## The one-line version No threshold is correct in isolation; it is correct relative to an alarm budget, a fleet size, a shift size worth catching, and the cost asymmetry between a false page and a slow detection. Say that, show the curve, and name fleet multiplication — that is the principal-level answer.