If the rule is "page on symptoms, not on causes", when is it nevertheless correct to page on a cause signal such as a filling disk or an expiring TLS certificate?

## Why the rule needs an exception at all Symptom-based paging is right because it maximises coverage and protects the responder's attention. But it has one structural weakness: it is inherently *reactive*. A symptom alert fires when harm has already begun. For most failures that is fine, because the mitigation is fast — roll back, fail over, shed load — and the difference between detecting at minute zero and minute two is small. It is not fine when the harm, once begun, is either irreversible or takes far longer to undo than to prevent. That is the entire justification for the exception, and it is worth stating in that form rather than as a list of blessed metrics. ## The three-part test **1. Is impact certain, not merely possible?** A certificate expiring at 02:00 will break TLS handshakes at 02:00. A disk filling at a steady 2 GB/hour with 6 GB left will be full in three hours. Contrast with CPU at 85%, which may stay at 85% indefinitely. Certainty is what turns a cause into a symptom you happen to be seeing early. Note that a projection is only as good as its assumptions: a fill-rate projection assumes the rate holds, so make the projection window short enough that the assumption is defensible and prefer a trend over a static threshold, since 10% free means very different things on a 100 GB volume and a 10 TB one. **2. Is the lead time comparable to the fix time?** This is the decisive part and the one candidates skip. Compute both sides. If the certificate renewal requires a change approval that takes two hours and you find out six hours before expiry, you are inside the danger zone and that is a page. If automated renewal runs daily and failure is detected thirty days out, that is a ticket — and if it is still failing at seven days, *then* it becomes a page. Escalating a condition from ticket to page as the runway shrinks is the right shape for these alerts. **3. Is the failure worse after impact than before?** Some conditions are cheap to fix while healthy and brutal afterwards. - A full disk on a database can stop the write-ahead log, corrupt state, and block the log rotation or compaction you would use to free space. You may not be able to log the incident on the box that is having it. - Losing quorum in a consensus system is not recoverable by adding a node once the cluster is already down. - A queue with no consumer may cross a retention boundary and drop data permanently, which no amount of later capacity restores. Where the cost curve has that cliff, paying the price of an early page is rational. ## Loss of redundancy The redundancy case deserves separate mention because it is the most commonly missed one. A service designed to survive one failure (N+1) has, after one failure, no margin left. Users see nothing. Every symptom signal is green. But the system is now one event away from an outage and it will stay that way until someone restores the spare. Whether that pages depends on the same arithmetic: how long does restoring the redundancy take, how likely is a second failure in that window, and how bad is the outage if it lands? Losing one of nine stateless replicas that auto-heal in ninety seconds is a ticket. Losing one of three zones, or one of three members of a consensus group, is a page. ## Discipline that keeps the exception from eating the rule The risk with any exception is that every team decides its favourite threshold qualifies. Three practices keep it honest: - **Write the justification into the alert.** The alert's own documentation should state the certainty argument, the lead time, and the action the responder takes in the next few minutes. If nobody can write the action, the condition is not a page. - **Prefer a deadline to a threshold.** "Disk will be full in under one hour at the current rate" is a better paging condition than "disk above 90%", because the first encodes lead time and the second encodes only a level. Same for certificates: page on days remaining, not on existence. - **Review the list.** Cause pages should be a small, named set — typically a handful per service — reviewed periodically against what actually fired and what the responder actually did. A cause page that fired five times and produced no action in any of them has failed its own test. ## What to say in an interview Give the rule, then the test, then a worked example with real arithmetic: "disk fills in 25 minutes, the fix is to expand the volume or purge logs, and that takes ten to fifteen — I want a human on it now, not after the writes start failing." The arithmetic is what shows you have made this call under pressure rather than read the principle.