How do you set a service's drain budget against the platform's stop timeout, and is a timed-out drain worth paging on?

## Two decisions wearing one name "How long should we drain for" hides an arithmetic constraint and a policy choice, and they should be argued separately. ### The arithmetic The platform sends a termination signal and kills the process after a fixed grace period. Everything the program does must fit inside that, not just the drain: - signal delivery and the shutdown path starting; - stopping intake and any deregistration from routing; - the drain itself; - acknowledging the last completed items, flushing buffered telemetry, closing files; - returning from main. So the budget is the grace period minus all of that, with margin. A budget set equal to the grace period is the failure mode that looks fine in review and never works: the kill lands at approximately the instant the timeout branch would have run, so you get neither the finished work nor the report about the unfinished work. If the two numbers live in different repositories — the budget in the service, the grace period in the deployment configuration — say so explicitly in both places, because a platform-side reduction silently breaks every service that was tuned to the old value. ### The policy What should happen when the budget runs out is not a Go question, it is a question about what an item is worth: - **Redeliverable and idempotent work.** The queue redelivers anything not acknowledged, and reprocessing is harmless. An incomplete drain costs some duplicated effort. This is an accepted loss: emit the metric, alert when the rate crosses a threshold, do not wake anyone at three in the morning for one instance. - **At-most-once side effects.** The item sends money, emails a customer, or writes something that is not naturally idempotent. Now an interrupted item is either a duplicate or a silent drop, and that is a correctness event worth paging on. The better engineering answer is usually to remove the ambiguity — make the effect idempotent with a key — but until that exists, the alert has to reflect the real cost. - **Work with an external commitment.** An item holding a lease or a lock elsewhere may leave that resource unavailable until it expires. The drain policy has to account for the recovery time, not only for the item. ## Why longer is not safer The instinct after one incomplete drain is to raise the number. Push back on it with the costs that are easy to forget: - **Rollback time.** Deploys stop instance by instance. A generous budget multiplies through the whole rollout, and the number that really matters during an incident is how fast you can get back to the previous version. - **Capacity during deploys.** Instances that are draining still hold memory, connections and downstream capacity while contributing nothing. - **It hides the real bug.** Nearly every persistent drain timeout is one unbounded operation. A longer budget converts a loud, reproducible failure into a slow deploy that nobody investigates. The honest sequence is: dump what was still running, bound that operation, then reconsider the budget — usually downward. ## When the work genuinely does not fit Some items legitimately take longer than any sane grace period. The answer is structural, not numerical: - **Make the unit of work smaller.** Split a long item into steps that each fit comfortably, so an interrupted item loses one step. - **Checkpoint.** Record progress so redelivery resumes rather than restarts. - **Stop taking long items before the drain.** If the arrival of a long item is predictable, stop accepting them earlier than the ordinary ones, so the drain begins with only short work in flight. - **Split the service.** If one class of work needs a fundamentally different shutdown profile, it probably wants its own deployment, with its own grace period, rather than dragging the whole service's deploy latency up. ## Who can overrule whom The grace period is usually a fleet-wide platform setting; the drain budget is the service's own. That gives the platform team the last word on the outer bound and the service owner the last word inside it. The productive framing when they conflict is not "give us more time" but "here is the distribution of item durations, here is what an abandoned item costs us, and here is what it would take to fit". A special-cased longer grace period for one service is a real cost paid by the people who operate the fleet, and it should be justified with numbers rather than asserted. ## Write it down The artefact this decision should produce is short and explicit: the budget, the grace period it fits inside, what an abandoned item costs, and whether the metric alerts or pages. Without it, the number is re-derived by whoever last had an incident, and the alerting posture drifts to whatever felt right during the last outage.