A ledger write replica is asked to stop, then forcibly killed 30 seconds later - what is that grace period for, and what happens when it expires?

## Stopping is two events, not one A container is never just "stopped". Two separate things happen, and the distance between them is the whole subject. 1. **The stop request.** The platform tells the workload to end. That is all it does. The process keeps running, keeps its memory, keeps its open connections, and keeps whatever work it had in progress. Nothing has been removed yet, and the platform is now waiting. 2. **The forced kill.** When the timer expires, the platform ends the process without asking. There is no handler for this, no callback, no final flush, no chance to write one more record. The process simply stops existing. The interval between the two is the **grace period**. Platforms let you declare it per workload and ship a default in the tens of seconds, because a default that is too generous makes every replacement slow and a default of zero makes every replacement lossy. The exact number is a policy choice you own, not a property of the workload. ## What the budget is for During the window the replica is supposed to spend the time, in this order: - **Leave the routing set**, so that whatever directs traffic stops choosing it as a target. - **Keep serving briefly**, because that removal does not take effect everywhere at once. - **Stop accepting new work** once the removal has had time to propagate. - **Finish the work it already accepted** - the requests in its handlers, the writes it has opened. - **Release what it holds** - claimed messages on a work queue, an exclusive lease, a reserved slot. - **Exit**, ideally well before the timer runs out. All of that has to fit inside the budget. If it does not fit, the tail of the list is what gets cut. ## What the expiry actually costs | State at the moment of the forced kill | What happens to it | |---|---| | A request already answered | Unaffected; the response left before the kill | | A request still inside a handler | Cut; the caller sees a broken connection, never a response | | An open, uncommitted write | The store rolls it back once the connection drops | | A committed write whose follow-on step never ran | Stays committed; the follow-on never happens | | A claimed message on a work queue | Stays claimed until its own claim timeout expires | | Buffered output not yet flushed | Lost with the process | The row that surprises people is the fourth. A single transaction against a durable store is safe: the store notices the dropped connection and rolls the transaction back, and the ledger is consistent. What is *not* safe is work that was never one transaction - a balance committed here and a notification that was supposed to be published there, a first step applied and a second step that the kill pre-empted. Graceful shutdown exists mostly to protect multi-step work, because single-step work was already protected by the store. ## The window is a ceiling, not a delay A common misreading is that declaring a 30-second window means every replacement takes 30 seconds. It does not. The workload exits as soon as it is genuinely finished, and the platform proceeds immediately. The only part of the window that is paid every single time is the deliberate pause you take while your removal from the routing set propagates - typically a handful of seconds. Everything after that is spent only if there is real work to spend it on. That asymmetry is why a generous window is cheaper than it looks and why a short one is more expensive than it looks. A window that is too short does not save you time on the common case, where work finishes quickly anyway; it only costs you the rare, long, important unit of work. ## Why interviewers ask this Because the failure is invisible in a healthy system and obvious in a post-incident review. Scale-downs, rollouts and node replacements all end with some replica being asked to stop, and the ledger's mysterious gaps - a debit with no matching credit, a request the client swears it sent - line up exactly with those events. A candidate who treats the stop request as "the container stops" has no model for the gap. A candidate who can name the two events, the budget between them, and the fact that the second one cannot be handled, can reason about every one of those incidents from first principles, on any platform.