Incident Management

Roll back to the last known-good release first, then investigate. Restoring users is the goal during an incident, the deploy timing is strong enough evidence to act on, and reading logs leaves users broken for however long the debugging takes.

Incident updates go out on a fixed cadence tied to severity — commonly every 30 minutes at the top severity — and each one states user-visible impact, what the response is doing now, and the time of the next update. Never promise a restoration ETA.

The Incident Commander steers, the Operations Lead is the only one changing production, the Communications Lead handles everyone outside the response, and the Scribe timestamps decisions and actions. The IC stays out of debugging because attention is single-threaded — in a stack trace, nobody is steering.

Grade on observable user impact: what share of users or requests is affected, how critical the blocked journey is, whether a workaround exists, and whether data or money is at risk. Which component failed predicts none of those.

Match the lever to the most likely change vector — code, config, a dependency, traffic, or capacity — then prefer whichever is fastest to take effect, smallest in blast radius, and easiest to undo. Apply one lever at a time so the user-facing signal tells you which one worked.

The Incident Commander owns the response, not the fix: they hold the current picture, assign every task to a named person, and make the calls. Deep technical knowledge is not the qualification — command is a coordination job a trained responder holds.

An alert is an automated signal that some condition was met. An incident is a human declaration that real or suspected impact warrants coordinated response, with a severity, an owner and a written record. Triage is the step between them.

Capture timestamped entries live for observations, actions and decisions, tagged as which of the three they are, plus the four anchor times: detection, declaration, mitigation applied, impact ended. Reconstruction fails because chat records when something was mentioned, not when it happened, and memory rewrites what people knew.

Capture only what dies with the process — thread stacks, heap state, local files, in-memory queues — and prefer quarantining one failing instance out of rotation over restarting them all. Anything already shipped to central logging, metrics or tracing survives the mitigation, so do not wait for it.

Verify against the user-facing SLI at the granularity that failed, not against a calmer dashboard or a cleared alert. The classic traps are an error ratio that fell because traffic fell, a metric window that has not turned over yet, and a backlog still draining behind a healthy-looking front door.

Escalate on a clock, not on exhaustion: set a time box when the incident starts and pull in the specialist or vendor when it expires, even if you might have solved it yourself. Effective escalation keeps ownership local, runs in parallel with mitigation, and arrives with evidence attached.

Public posts are triggered by confirmed customer-visible impact, not by a known cause. They state the symptom, scope and next update time in plain language with no internal names or speculation, while the internal update stays candid about hypotheses, ruled-out theories and mitigation options.

Fix forward when rollback is impossible, ineffective, or slower than the fix: an irreversible migration has already run, the previous build carries the same defect, or reverting takes forty minutes while a one-line change takes four. Bound it with a hard deadline and a cruder fallback mitigation.

Hand off explicitly, never by drift: brief the incoming IC on impact, timeline, active workstreams and their owners, and pending commitments; state "you are now IC" in the channel; then the outgoing IC shadows briefly and leaves. Fatigued judgment costs more than the context reload.

One person holding every role means the response has no coordination, no timeline, and a bus factor of one: all state is in their head. Take command, get a spoken state dump, write it down, split the work among named owners, and put someone between them and the executives.

Declare now. The costs are asymmetric: an over-declaration is undone with one message and a few interrupted minutes, while a late declaration adds unmitigated impact plus the response ramp-up you could have run in parallel. Declare low and adjust.

Upgrade immediately on discovery, not at the next sync, and restate the grade explicitly with the time. Downgrade only after impact has actually stopped and recovery is verified — never to quiet the response — and record the peak severity, not the closing one.

Give each audience its own surface with one publishing point: a responder working channel, a broadcast status channel on a fixed cadence, and a named human who briefs leadership directly. Agree notification thresholds by severity in advance so nobody negotiates access during the incident.

Attack the two things that fill those forty-two minutes: decision latency and lever latency. Give each service a short menu of mitigations with measured times, pre-authorise on-call engineers to pull them without approval, and treat a slow rollback or an undrilled failover as a defect to fix.

Build a trained, cross-team pool of commanders rather than defaulting to service owners, qualify them by shadowing then leading under supervision, and have leadership grant the authority in writing beforehand: an IC can pull in anyone, suspend other priorities, and override objections for the incident's duration.

Treat it as an incentive problem, not a wording problem. Diagnose whether teams are inflating or deflating, calibrate with worked reference incidents rather than longer definitions, re-grade a sample of closed incidents regularly, and decouple punishing process from high tiers.