A responder followed a runbook's "restart the service" step for a symptom that resembled the documented one but had a different cause, and made the outage worse. How prescriptive should a runbook be, and what has to guard a copy-paste command?
answer
- not prescriptive versus thinking — gated versus ungated
- every command carries a condition
- reversible steps can move fast
- name the exit before someone needs it
- irreversible means a second pair of eyes
basics
~20 sBe maximally prescriptive about mechanics and explicit about the conditions under which each step is valid. Every copy-paste command needs a stated precondition, its blast radius, whether it is reversible, and an escape hatch that says stop and escalate when the symptoms do not match.
solid answer
~50 sThe dichotomy is false — the useful axis is not "prescriptive versus thinking" but "where the branch points are". Mechanical steps should be exact and copy-pasteable: at 3am, retyping a command from a description is how typos take down the wrong cluster. What guards them is the gate around each one — the precondition that must hold, the blast radius, and the reversibility. Where a genuine judgment call exists, the runbook should surface it as an explicit branch with the evidence needed to choose ("if queue depth is growing, do A; if it is flat and latency is high, do B") rather than either hiding it inside a linear script or hand-waving it as "investigate". And every runbook needs the escape hatch: if what you see does not match the described symptoms, stop and escalate rather than continuing down the list. Irreversible steps get a second person or leave the runbook entirely.
go deeper
Know that a runbook step is not unconditionally safe: before running a command you should check the stated precondition and, if what you see does not match the description, stop and ask for help rather than continuing.
Explain the three things that travel with every command — precondition, blast radius, reversibility — and why an explicit branch with distinguishing evidence beats both a linear script and vague advice to investigate.
Demonstrate the allocation judgment: where to spend gates given that each one slows mitigation, how you handle destructive steps, and how a misapplication turns into a same-shift repair rather than an aging action item.
Own the standard that decides what a solo responder is permitted to do unaided at all: which classes of action must never appear in a runbook without a second person, and what that confirmation requirement costs in mitigation time across every incident.
## Why both extremes fail A purely prescriptive runbook — a linear list of commands with no gates — optimises for speed and gets executed blindly. That is fine while the failure matches, and dangerous the moment it does not. The scenario in the question is the canonical case: a restart clears a wedged worker, and the same restart on a node that is crash-looping because a bad config is being loaded at startup does nothing, or replaces a partially serving instance with a fully dead one. A purely judgment-based runbook — "investigate the queue, determine the cause, apply appropriate mitigation" — is not a runbook. It offers nothing to a responder who has never operated the service, which is the reader it exists for. It is also unfalsifiable: it can never be found wrong, and therefore never gets fixed. ## The real axis: gated steps What separates a safe runbook from a dangerous one is not how much prose surrounds the commands, it is whether each action carries the conditions that make it valid. **Precondition.** State what must be true. "Run this only if the pods are Running but not serving; if they are in CrashLoopBackOff, skip to section 4." A precondition converts a command into a conditional, which is what the author actually meant when they wrote it. **Blast radius.** Say what the command touches. One instance or the fleet? In-flight requests dropped or drained? A responder from another team cannot infer this, and the difference between restarting one worker and rolling the whole deployment is the difference between a blip and an outage. **Reversibility.** Say whether it can be undone and how. Reversible steps can be taken quickly on partial evidence; irreversible ones cannot, and that asymmetry should be visible at the moment of decision. ## Where judgment belongs, and how to write it Some decisions genuinely cannot be predetermined, and the runbook's job then is to make the decision *explicit and informed* rather than to pretend it does not exist: - Present it as a branch, not as advice. "If A, go to 3.1. If B, go to 3.2. If neither, escalate." - State the evidence that distinguishes the branches, with where to look and what the values mean. - Do not leave a branch dangling. Every arm ends in either a step or an escalation. This structure is worth more than it looks: it tells the responder what the author was actually worried about, which is often the fastest route to understanding the failure. ## The escape hatch Every runbook needs an explicit instruction covering the case the author did not anticipate: *if the symptoms do not match what is described here, stop and escalate rather than continuing.* Without it, a responder who has already followed four steps feels committed and keeps going — the sunk-cost pull is strong at 3am, and "the runbook told me to" feels like authorisation. Naming the exit removes the social cost of taking it. ## Destructive steps The boundary case is a step that cannot be undone: deleting a queue, failing over a primary database, truncating a table, wiping a cache the whole fleet depends on. Options, in rough order of preference: 1. Leave it out of the runbook and escalate to the service owner at that point. 2. Keep it, but gate it behind a second person's explicit confirmation. 3. Keep it with a hard precondition and a documented recovery path, if the failure it addresses is time-critical enough that waiting for a second person costs more than the risk. The principle behind all three: a solo, half-awake responder following instructions correctly should not be able to cause permanent damage. ## The cost you are trading Every gate slows execution, and mitigation speed is the reason the runbook exists. A runbook where every line carries three qualifiers is unreadable, and unreadable means skimmed. So spend the gates where the cost of being wrong is high — wide blast radius, irreversible, or applicable to more than one cause — and let the cheap reversible steps stay terse. That allocation is the actual skill, and an interviewer asking this question is usually listening for whether you make it consciously. ## Feeding the fix back When a runbook is misapplied, the outcome is not just an incident — it is information about a missing gate. The precondition that would have stopped it should be added by whoever hit it, in the same shift, rather than filed as an action item that ages. That is the mechanism by which runbooks get safer with use instead of merely older.
- How do you decide which steps deserve a precondition and which can stay terse?By the cost of being wrong. A step that is reversible, narrow in blast radius, and valid for every cause of the symptom can be a bare command. A step that is irreversible, fleet-wide, or only correct for one of several possible causes needs the gate. Qualifying everything equally makes the document unreadable, and unreadable documents get skimmed.
- After a runbook is misapplied during an incident, what should happen to the runbook?The missing precondition gets added by the person who hit it, ideally in the same shift while the detail is fresh. Treating it only as a postmortem action item means it ages in a backlog while the same trap stays armed. The incident is evidence of a specific defect in a specific document, and that is the cheapest kind of fix to make.
saying these in an interview costs you the question
- Says runbooks should just list commands so nobody has to think
- Writes "investigate and mitigate appropriately" as a step
- Includes irreversible commands with no gate or approval
- Assumes matching symptoms mean matching causes
- Leaves a decision branch with no escalation arm