How would you structure a primary/secondary on-call rotation, and what does adding a secondary actually cost the team?

## What the secondary slot is for A secondary exists so that a page never falls on the floor and so a primary who is deep in an incident is not simultaneously the person answering everything else. In practice a well-defined secondary owns three things: pages the primary did not acknowledge; being the second responder when an incident is declared, so the primary can debug while someone else handles coordination or data-gathering; and absorbing non-urgent requests and tickets during the shift so the primary's attention stays on production. How the page actually reaches the secondary — acknowledgement timeouts, escalation tiers, fallbacks — is escalation policy design and a separate subject. What belongs to rotation design is who sits in that slot, how often, and at what cost. ## Three ways to staff it **Same pool, offset by one shift.** The most common arrangement: this week's secondary becomes next week's primary. It has a real side benefit — the incoming primary spent the previous shift adjacent to every incident, so they arrive with context and the handoff is shorter. It is also the most expensive in commitment terms. **A separate expert or manager backstop pool.** A small group of senior engineers or the manager sits behind every primary. This keeps the main rotation's commitment at one slot per cycle, but concentrates load on a few people and can entrench single points of knowledge. It works best when the secondary is genuinely rare — a true last resort rather than a routine second responder. **A shared secondary across teams.** For a platform with several small owning teams, one secondary slot can back several primaries. Cheaper per team, but the secondary needs breadth, so the runbooks have to carry the load. ## The arithmetic you must say out loud If one pool of N people covers both slots on week-long shifts, each person is on call `2 x 52 / N` weeks a year: - N = 8 → about 13 weeks a year, one primary and one secondary week every eight - N = 6 → about 17 weeks a year, roughly a third of the year touching the pager That is why a secondary is not a free safety improvement. In a six-person team, adding a secondary from the same pool is a bigger change to people's lives than most of the reliability work it is meant to protect. Either the rotation is deep enough, or the secondary comes from somewhere else, or you do without one and rely on a broader escalation path. ## Keeping it from becoming a second primary A secondary that is paged for most incidents is not a backstop; it is a second person on the same shift with half the authority. The signals are easy to read: the secondary's page rate approaches the primary's, or the secondary is routinely the one who resolves things. When that happens the cause is usually one of two, and they need different fixes. Either the primary rotation is not functioning — someone is not acknowledging, or is not equipped to act — which is a training or staffing problem. Or the escalation timing is so aggressive that the secondary is effectively paged in parallel, which is a policy problem. A rough expectation worth stating: the secondary should be paged for a small fraction of what the primary handles, and if it is not, something upstream is broken. ## Do not confuse secondary with shadow A shadow is a third arrangement entirely: someone learning the rotation who receives pages for exposure but carries no responsibility for resolving them, with a real primary always on the hook. Shadowing is a training mechanism and it does not add coverage. Counting a shadow as a secondary is how teams convince themselves they have depth they do not have. ## When to skip the secondary If the rotation is thin, incidents are rare, and the escalation path can reach a wider group quickly, a formal secondary may cost more than it returns. The honest version of that decision names the risk being accepted — an unacknowledged page takes longer to reach a human — and pairs it with something concrete, such as a shorter escalation timeout and a documented group fallback.