You join a team whose pager delivers roughly 50 alerts a week, and most are acknowledged with no action taken. Describe the process you would run over the next month to reduce that, and how you would decide the fate of each rule.

## Why this question is asked Every SRE interview reaches some version of it, because it separates people who have carried a pager from people who have read about one. The weak answer is a list of techniques. The strong answer is a *process* with an order, evidence at the front, and an explicit account of what could go wrong. 50 pages a week across a rotation is roughly one every three hours around the clock. No human sustains that, and the damage is not only fatigue: a pager that is mostly wrong trains its recipients to acknowledge without reading, so the genuine page gets the same reflex as the other forty-nine. ## Step 1: measure before you touch anything Do not start by deleting the rule that annoyed you last night. Build the table: | per rule | why it matters | |---|---| | firings in 60–90 days | finds the volume leaders — noise is almost always concentrated in a handful of rules | | pages vs non-paging notifications | separates 'loud' from 'merely present' | | median time to acknowledge | very fast acks with no follow-up work is the signature of reflex dismissal | | share that led to a human action | the column that decides disposition | | clustering in time | distinguishes one fan-out event from a chronically noisy rule | The action column rarely exists already. Capture it going forward: at acknowledgement, the on-call marks the page actionable or not, optionally with one word of reason. It costs seconds, and after one full rotation you have real evidence rather than a debate about whose intuition is right. Distribution matters as much as totals — 50 pages a week is often three rules producing 40 of them, which means the first week of work is disproportionately valuable. ## Step 2: the disposition ladder Work the ranked list top-down. For each rule, in this order of preference: 1. **Fix the underlying system.** If a rule fires twice a week because a job legitimately runs out of memory twice a week, the honest fix is the memory, not the alert. This is the only disposition that reduces noise *and* improves reliability, and it is the one teams skip. 2. **Collapse the fan-out.** If one fault produces forty notifications, grouping on the right label subset or suppressing the downstream alerts behind their cause removes the volume without removing any signal at all. Cheapest real win available. 3. **Retune.** The rule is right but the timing or the number is wrong: a threshold set at normal peak behaviour, or a missing pending duration so it fires on transients. Retune and re-measure next rotation. 4. **Demote off the paging path.** The condition is worth recording and worth working during business hours, but nothing useful happens at 3 a.m. Move it to a queue that a human reads in daylight. 5. **Retire.** It has never produced an action, or another rule always fires alongside it and is the one people actually work. ## Step 3: the risk, and how to bound it The way this goes wrong is deleting the only detector for a rare but severe failure. It fired twice in three years, both times legitimately, and it looks like noise in a 90-day window. Bound it: **demote before you delete.** Move the candidate off the paging path for a quarter while keeping it firing and recorded, then check whether any incident in that quarter went undetected or was found by a customer instead of by an alert. If nothing was missed, retire it. Keep the pruning decisions themselves in version control with a reason attached, so a later postmortem can ask 'did we remove the alert that would have caught this?' and get an answer. Also track the counter-metric the whole time: **what fraction of incidents were detected by an alert rather than reported by a user?** Noise reduction that quietly drives that number down has made things worse, not better, and it is the single check that keeps this work honest. ## Step 4: make it recurring, or it decays A one-off cleanup regrows within two quarters. Anchor it to something that already happens: - **At every handoff**, the outgoing on-call presents the shift's pages and proposes a disposition for the noisiest one. This is the highest-leverage habit in the whole practice: it puts the change proposal in the hands of the person with the freshest evidence, at a meeting that already exists. - **A standing target.** Google's SRE book argues an on-call shift should handle at most roughly two incidents per 12-hour shift, on the reasoning that properly handling one — response, mitigation, write-up — consumes about six hours. Whatever number you adopt, having one turns 'the pager is bad' into a measurable, arguable state. - **A bar for new rules.** Any new page-severity rule states what a human is expected to do when it fires. If nobody can write that sentence, it is not a page. ## What to say in the interview Compress it: measure per-rule volume and actionability, attack the concentrated leaders first, prefer fixing the system over tuning the alert and tuning over deleting, demote for a quarter before retiring anything, watch the alert-detected share of incidents so you can prove you did not go blind, and hang the review on the on-call handoff so it does not decay.