Rolling back, restarting, or replacing instances during an incident usually destroys the state you would need to explain the failure later. What do you capture before you pull the mitigation lever, and how do you keep that capture from delaying the mitigation itself?

## Why this is a real tension, not a platitude Mitigation-first response has a genuine cost, and this is it. The fastest mitigations — restart, roll back, replace the instances, fail away from the region — all work by *destroying the state that is misbehaving*. That is precisely why they work, and it is also why the postmortem sometimes opens with "we never found out what happened." A candidate who has actually carried a pager knows this tradeoff exists and has a cheap answer to it. A candidate who has only read about incidents either forgets evidence entirely or, worse, spends fifteen minutes collecting it while users fail. ## Sort the evidence by whether it survives The useful mental split is durable versus ephemeral. **Durable — do not wait for it.** Anything that has already left the process is safe: log lines shipped to a central store, metrics scraped or pushed to a time-series backend, traces exported to a tracing backend, load-balancer and proxy access logs, audit records of deploys and configuration changes. All of it will still be there after the rollback. Collecting it during the incident buys nothing and costs minutes. **Ephemeral — dies with the process or the node.** This is the list worth spending seconds on: - **Thread stacks.** What are the threads actually blocked on? This is often the whole answer for a hang or a deadlock, and it is unrecoverable once the process exits. - **Heap or memory state.** For a suspected leak or an out-of-memory condition, a heap snapshot is the only artifact that shows what was actually retained. It is also the most expensive capture on this list, and may itself stall the process — take it from an instance that is already out of rotation. - **In-memory and local queues.** Depth, oldest entry, and whether anything is being dropped. - **Node-local files.** Crash artifacts, core dumps, or logs that were never shipped because the shipping agent was itself failing. - **Live connection and socket state.** Whether the process is holding connections open to a dependency that has stopped answering. ## The specimen: the highest-value habit here The technique that resolves the tension almost entirely is **quarantine one, mitigate the rest**. Remove a single affected instance from the load-balancer or service-mesh rotation, or scale the fleet by one and leave the sick one running, then roll back or restart everything else. Users are recovered on the healthy fleet within the normal mitigation time, and you keep a live, still-broken specimen you can attach a debugger to, dump repeatedly, and reason about at leisure. This costs one instance's worth of capacity and roughly the same number of seconds as the mitigation itself. Note the constraint that makes it fail: if the failure is *caused* by the specimen — it holds a distributed lock, it is poisoning a shared cache, it is hammering a dependency — quarantining it does not stop the impact and you must kill it. Judgment, not ritual. ## The timebox is the actual rule Evidence collection expands to fill the incident if you let it. The bound that experienced responders apply is roughly: **if it takes seconds and requires no thinking, do it; otherwise mitigate now and accept the loss.** Concretely, a thread dump on one instance is fine. Deciding which of four possible diagnostic tools to attach, and to which instance, is not — that is diagnosis wearing an evidence-collection costume. The way to make more evidence survivable is to move that work out of the incident entirely: continuous profiling, automatic heap dumps on out-of-memory conditions, log shipping that keeps up under load, and retaining the previous generation of instances briefly after replacement. Every one of those turns an in-incident decision into a decision you already made, calmly, months ago. ## The evidence nobody thinks of The most commonly lost artifact is not a dump. It is **the timeline**: what was observed at what time, what action each responder took, at what time, and what happened next. Human memory reconstructs this badly and confidently. Two habits fix it: announce every action in the incident channel *before* taking it with a timestamp, and snapshot the graphs you are looking at now — dashboards are frequently rebuilt, and retention windows and downsampling can quietly coarsen the exact minutes that mattered by the time someone opens the postmortem a week later.