You are configuring the escalation policy for a new customer-facing service in a paging tool such as PagerDuty or Opsgenie. What rungs would you define, how long would you set the acknowledgement timeout, and what must the final rung do?

## What an escalation policy is An escalation policy is the ordered list of who gets notified when an alert fires, and how long the system waits at each step before giving up on that person and moving to the next. The alert itself decides *that* a human is needed; the escalation policy decides *which* human, and what happens when that human does not answer. In tools like PagerDuty or Opsgenie this is a first-class object attached to a service, so every alert routed to that service inherits the same chain. The whole design turns on one number and one guarantee: the acknowledgement timeout, and the terminal rung. ## The rungs A standard chain for a customer-facing service looks like: 1. **Primary on-call for the owning team.** The person who is holding the pager for this service right now — targeted by schedule, never by personal name, so the policy keeps working when the rotation moves. 2. **Secondary on-call**, after the ack timeout expires. The secondary exists precisely for the case where the primary is unreachable — asleep through a phone on silent, in a tunnel, or already deep in another incident. 3. **Manager or named fallback team**, after another timeout. This rung is not there to debug. It is there because someone with authority needs to notice that a customer-facing service is broken and nobody has picked it up, and then go find people — call phones, wake the team, pull in another team. Two or three rungs is normally the right depth. Every rung is minutes of unmitigated outage, so a policy with five rungs is not more robust — it is a policy where the first four targets are decorative and the real answer arrives half an hour late. ```yaml # rung targets are schedules and teams, never individual people rungs: - target: checkout-primary-oncall # schedule ack_timeout_minutes: 5 - target: checkout-secondary-oncall # schedule ack_timeout_minutes: 10 - target: checkout-engineering-manager ``` ## Choosing the acknowledgement timeout The timeout is the time you are willing to lose to a human who is not answering. For a high-urgency page on a customer-facing path, five minutes is a common choice: it is roughly how long it takes a woken engineer to get to a laptop and press acknowledge. Ten to fifteen minutes is reasonable for lower-urgency services where the cost of a delayed response is small and the cost of unnecessarily waking a second person is not. Get it wrong in either direction and it costs you. Too short, and you burn your secondary on pages the primary was already handling — that is real pager load spent on nothing, and it trains the secondary to ignore escalations. Too long, and a single phone on silent turns a five-minute mitigation into a twenty-minute outage. If you are consistently escalating past rung one, the problem usually is not the timeout; it is that the primary is overloaded or the notification never actually arrived. ## Delivery redundancy inside a rung Escalating to another person is the coarse fallback. Inside a single rung, most tools let you configure a notification ladder for the individual: push notification immediately, SMS a minute later, phone call after that. This matters because the dominant failure is not "the engineer refused" but "the notification never landed" — a push that died with the app in the background, a carrier that dropped the SMS. A phone call that overrides do-not-disturb is what actually wakes people, so any high-urgency rung should reach a call before it escalates. ## The terminal rung The single most common defect in real escalation policies is a chain that ends in nothing. Every rung times out, the tool marks the incident unacknowledged, and no further notification is ever sent — the page is now a row in a database that nobody is looking at. Guard against it two ways: configure the policy to repeat the whole chain a fixed number of times rather than stopping after one pass, and make the last rung a target that is contractually reachable — a duty manager, a follow-the-sun partner team, or a broad team-wide notification. If your final rung is a single named individual, your escalation policy has a single point of failure with a personal life. ## Verify it before you rely on it An escalation policy is a control, and an untested control is a belief. Before a new service takes production traffic, fire a real test page and watch it walk the chain — wrong phone numbers, a schedule with a gap at the weekend, a rung pointing at a team that no longer exists, and a notification rule that only sends email are all things you discover at 3am otherwise. Re-test after any reorg, because the reorg is what breaks routing.