Why does a Go worker's shutdown path wait for in-flight work under a context.WithTimeout budget instead of waiting indefinitely?

## What a drain budget is A long-running Go worker that consumes a queue is not stopped instantly when it is taken out of rotation on a deploy. The platform sends a termination signal, waits some fixed grace period, and then kills the process outright. That window is the only time the program has to end tidily. *Draining* is what the process does with the window: stop pulling new items off the queue, let the items already in flight run to completion, then return from `main`. A *drain budget* is a wall-clock bound on the second step — normally a `context.WithTimeout` created at the moment shutdown begins, with a duration chosen to fit inside the platform's grace period. ```go drainCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second) defer cancel() select { case <-workersDone: // closed once the last in-flight item is acked log.Print("drain complete") case <-drainCtx.Done(): log.Printf("drain incomplete: %d items still in flight", inflight.Load()) } ``` ## Why the wait must be bounded The grace period exists whether or not your code respects it. If you wait forever for in-flight work, you have not removed the deadline — you have only moved it out of your program and into the platform's kill. The consequences are all bad ones: - **You lose the exit path.** A kill does not run your final log line, your metric flush, or any last write. Everything you wanted to know about the shutdown dies with the process. - **One straggler decides for everyone.** In-flight work usually means outbound calls, and an outbound call with no deadline of its own can hang for minutes. A single stuck item holds every other goroutine's cleanup hostage. - **You cannot tell success from failure.** A drain that finished cleanly and a drain that was killed halfway look identical from the outside. With a budget, "we abandoned 4 items" is a number you can graph and alert on. - **Deploys get slower and less reversible.** Rolling out or rolling back a fleet means stopping every instance. An unbounded drain makes the worst instance set the pace of the whole rollout. ## Stopping intake comes first A budget only helps if the in-flight set actually shrinks. The first act of the shutdown path is to stop accepting new work — stop the fetch loop, stop the poller, refuse new items — and only then wait. If intake is still running, the drain waits for a set that keeps refilling and will always hit the deadline. ## What the deadline branch is for The point of the budget is not that the wait ends; it is that *you* get control back when it ends. In the timeout branch you can: - count and log the abandoned items, ideally with their ids, so someone can check whether they were redelivered; - increment a metric so "drains that ran out of budget" is a rate, not an anecdote; - dump a goroutine profile so the straggler's stack is on record; - exit with a status that distinguishes an incomplete drain from a clean one. ## The budget is not a delivery guarantee A bounded drain reduces the number of items interrupted mid-flight; it never removes the possibility. Anything that must not be lost still needs redelivery on the queue side and handlers that tolerate seeing the same item twice. A team that treats "we drain gracefully" as a substitute for at-least-once redelivery has simply made the loss rarer and harder to notice. ## Choosing the number The budget must be strictly smaller than the platform's grace period, with slack for signal delivery, the final flush and the exit itself. Beyond that the number is a service-level judgment: long enough that ordinary items finish, short enough that a deploy or a rollback is not held up by the slowest instance. If typical item processing time does not fit in a sane budget, the fix is to make the unit of work smaller or checkpointable, not to keep raising the number. ## Common mistakes - Deriving the budget from a context that is already cancelled, so the timeout branch fires immediately. - Setting the budget equal to or greater than the platform's grace period, so the process is killed before the timeout branch runs. - Waiting for the work but never stopping intake. - Logging "shutdown complete" on both branches.