SRE Practices

A good SLI tracks what a user experiences — success, latency, freshness — and moves with user happiness. CPU and cache hit rate are internal resource metrics: they spike while users are fine, and stay flat while users suffer.

An error budget is one minus the SLO target over the window. A 99.9% target across 30 days allows about 43 minutes of unavailability, or 0.1% of requests, before the objective is missed. Spending it means shipping change against that allowance.

An SLI is a measured number, usually good events divided by valid events. An SLO is the internal target that number must meet over a defined window. An SLA is the customer-facing contract, set looser than the SLO, and it is the only one carrying penalties.

Burn rate expresses error-budget consumption as a multiple of the sustainable pace: rate 1 spends the whole budget exactly at the window's end. Compute it as the observed bad-event ratio divided by (1 minus the SLO target).

Define latency as the proportion of valid requests served faster than an explicit threshold — for example 99% of checkout requests under 400 ms. An average hides the slow tail entirely: a handful of thirty-second requests barely move it while those users are the ones leaving.

Latency, traffic, errors and saturation. Latency and errors are what users feel and are the natural paging signals; a traffic collapse is also user-visible. Saturation measures resource fullness — a cause, so it belongs on dashboards and tickets.

Page only when a human must act within minutes and can actually change the outcome. Ticket when the work is real but can wait days. Log when nobody will act. Each tier is a response-time promise.

Deduplication collapses byte-identical alerts from redundant senders into one. Grouping batches alerts that share a chosen set of labels into a single notification after a short wait window. Grouping harder costs detection delay and can bury an unrelated failure inside an already-acknowledged page.

Symptom alerts fire on what users experience — failed requests, slow responses, a service level burning down. Cause alerts fire on machine state such as CPU or disk. Page on symptoms; send cause signals to tickets and dashboards.

Measure before changing: per rule, count firings over the last quarter and what share led to a human action. Then apply a disposition ladder — retire, demote off the paging path, retune thresholds and durations, group or suppress the fan-out, or fix the underlying instability — and make the review recurring at every handoff.

A per-incident channel gives one authoritative place for the response, so anyone joining reads the scrollback instead of interrupting responders, and it becomes the raw material for the postmortem timeline. A voice bridge is faster for fast-moving coordination but leaves no record, so decisions and actions still get typed into the channel.

Roll back to the last known-good release first, then investigate. Restoring users is the goal during an incident, the deploy timing is strong enough evidence to act on, and reading logs leaves users broken for however long the debugging takes.

Incident updates go out on a fixed cadence tied to severity — commonly every 30 minutes at the top severity — and each one states user-visible impact, what the response is doing now, and the time of the next update. Never promise a restoration ETA.

The Incident Commander steers, the Operations Lead is the only one changing production, the Communications Lead handles everyone outside the response, and the Scribe timestamps decisions and actions. The IC stays out of debugging because attention is single-threaded — in a stack trace, nobody is steering.

Grade on observable user impact: what share of users or requests is affected, how critical the blocked journey is, whether a workaround exists, and whether data or money is at risk. Which component failed predicts none of those.

Acknowledging a page stops the escalation chain and claims ownership: it tells the system a human is now working the problem. Acknowledging and then going back to sleep is worse than never answering, because nobody else will be notified.

Page the service's primary on-call first, escalate to the secondary after an unacknowledged timeout of roughly 5-15 minutes, then to a manager or a named fallback team. The final rung must reach a guaranteed-reachable human and never dead-end.

Measure pager load per shift, not per month: pages per shift, the share that arrive outside business hours, the share that turned out to be actionable, and hours lost to interrupts. Google's SRE book suggests at most two incidents per 12-hour shift.

A single-site 24/7 primary rotation needs roughly six to eight engineers, so each person carries the pager about one week in six. With fewer people, narrow what pages overnight, merge the pager with another team, or add a second site.

A handoff transfers state, not a summary: open incidents and their next step, every alert silenced during the shift with its expiry, in-flight changes and migrations, temporary manual fixes with an owner, and risky events scheduled in the next shift.

The trigger is the event that started this outage; the underlying cause is the latent condition that made the trigger dangerous. Removing the trigger prevents this instance, fixing the latent condition prevents the whole class — and they cost very different amounts.

A good action item names one concrete change, has a single named human owner, a priority and a due date, lives in the team's normal tracker, and has an unambiguous definition of done. Bad items are open-ended investigations or exhortations to be careful.

5 Whys follows one causal line and stops at the first fixable defect, so it names the bug but not the review, test, rollout and detection gaps that let the bug reach production. Ask "what else" at each step and record multiple contributing factors.

Blameless means the postmortem asks how the system let a reasonable action cause harm, not who to punish. Accountability survives as named owners for the fixes and an obligation to explain honestly; deliberate recklessness is a management matter handled outside the document.

"Human error" is where an investigation stopped, not an explanation. Treat it as a symptom and record what made the wrong command reachable, plausible and undetected — no confirmation prompt, identical staging and production shells, a stale runbook, time pressure — and fix those.

Toil is manual, repetitive, automatable, tactical work with no enduring value that scales with the service. Manual work failing those tests — a one-off migration, or novel debugging of a new failure — is engineering or overhead, not toil.

Closing the loop buys speed and consistency: the fix runs in seconds, identically, without waking anyone. In exchange you take on a new production actor that changes systems unsupervised, so it needs guardrails, its own telemetry, and a named owner.

A usable runbook states its trigger and the user impact, then gives diagnostics with expected output, remediation with preconditions and blast radius, a verification signal that proves the fix worked, an escalation path with a time-box, and metadata naming the owner and last validation date.

The watchdog suppressed the symptom without reporting it, so a slowly worsening fault stayed invisible until it outran the remediation. Auto-remediation must count every action, escalate when the action rate climbs, and refuse to keep repairing indefinitely.

Tie maintenance to use: whoever follows a runbook during a page repairs it in the same shift. Bind each runbook to the alert that links it so orphans are visible, review them at on-call handoff, stamp owner and last-validated date, and delete more than you write.

A load test applies expected peak traffic to confirm latency and error targets hold. A stress test pushes past that limit to find where and how the service breaks. A soak test runs moderate load for hours to expose slow leaks.

Organic growth is extrapolated from your own traffic history and arrives smoothly. Launch-driven demand has no history to extrapolate, so its size and timing must come from business inputs and be provisioned before the event, not corrected after it.

The knee is where achieved throughput stops tracking the offered rate and p99 latency starts climbing super-linearly — queues are no longer draining. The usable limit is the highest sustained rate that still meets the latency target, which sits below the knee, not at peak throughput.

Compute is usually the easy part. The long poles are cloud quota increases and capacity reservations for specific instance types and zones, stateful work like resharding or index builds, and third-party rate limits. Start those first and leave the final weeks for verification and a freeze.

Random shedding drops checkout traffic and background prefetches at the same rate, so it damages revenue-bearing work to save cheap work. Choosing requires a criticality label attached at the entry point and propagated to every downstream call, with each server shedding lowest tier first.

A game day is a scheduled exercise where a team induces or simulates a failure and responds to it as if it were a real incident. It tests the human path — detection, paging, runbooks, comms — which no amount of monitoring configuration proves on its own.

The steady-state hypothesis is a falsifiable claim that a measured, user-visible output of the system stays inside a defined band while a fault is present. State it on output the customer feels, not on internal resource metrics.

Start with dependency faults — blackhole each downstream one at a time to check that every call you call optional really is — then latency on the busiest ones, then losing one whole zone. Random pod kills come last because rolling deploys and node scale-down already kill pods daily.

An untested recovery time objective is an estimate, not a number. Verify it by executing a real failover with a clock running from detection to service restored, in a low-traffic window with a tested way back — then treat the measured time, not the plan's, as the truth.

Minimize along population, duration, severity and reversibility: smallest slice of traffic or one instance, a hard time cap, the mildest fault that still tests the claim, and an undo. Widen one dimension at a time only after the hypothesis holds.

A runtime feature flag turns recovery from a redeploy into a configuration change, cutting mitigation from tens of minutes to seconds. That only holds if the flag is evaluated per request, the old path still works, and the change reaches every instance quickly.

A traffic percentage is a random sample, not a representative one. A rare code path, a single tenant, or one old client version can send so few requests that the canary receives none of them. Choose the canary population deliberately, not just its size.

Only the code was versioned. Configuration applied from a moving HEAD stays at its new value, so half the release is still live after the rollback. Bind the artifact digest and the config revision into one versioned release record you revert as a unit.

Progressive rollouts miss anything that needs time, scale, a specific event, or a specific population: slow leaks and disk fill, dependencies that only saturate at full traffic, monthly batches, and one tenant or old client. Each needs its own control.

A renamed column leaves rolled-back code reading a column that no longer exists, and the reverse migration destroys data. Sequence it expand-contract: add the new column, dual-write, backfill, switch reads, and drop the old one only after the rollback window closes.