How would you design an asyncio service's shutdown to fit an orchestrator's grace period?

## The default is a policy, and it is rarely your policy Left alone, `asyncio.run()` cancels every pending task the instant your main coroutine returns, with no budget and no ordering. That is a fine default for a script and a poor one for a service, where "in flight" means a half-answered request or a batch that has been accepted but not yet durably stored. A production shutdown is something you design; the runner's teardown is the safety net underneath it. ## Four phases **1. Quiesce.** Install the handler on the loop, not with a raw signal handler: ```python loop = asyncio.get_running_loop() stop = asyncio.Event() for sig in (signal.SIGINT, signal.SIGTERM): loop.add_signal_handler(sig, stop.set) ``` The callback does nothing but set an event — no cancelling, no closing. Everything that follows is ordinary async code you can read and test. On waking: fail the readiness probe first so the load balancer removes you, then stop accepting new work (close the listener, stop pulling from the queue). Keep liveness green, or the supervisor will kill you for the crime of shutting down. **2. Drain with a deadline.** Wait for accepted work only, and bound the wait: ```python _, still_running = await asyncio.wait(in_flight, timeout=budget) ``` **3. Cancel the stragglers** and await them so their `finally` blocks actually run: `Task.cancel()` on each, then `asyncio.gather(*still_running, return_exceptions=True)`. **4. Finalize.** Return from `main()`. The runner cancels whatever you missed, finalizes async generators, joins the default executor, closes the loop. ## The budget arithmetic A supervisor typically sends `SIGTERM`, waits a grace period — 30 seconds is a common default — and then sends `SIGKILL`, which cannot be caught, handled or ignored. Everything you want to happen has to fit inside that window, minus a margin for the final flush and process exit. If your drain budget equals the grace period you will be killed mid-cleanup, which is worse than cancelling early, because a task that is cancelled at least gets to run its `finally`. One trap hides in the teardown itself: `asyncio.run()` joins the default executor with a **five-minute** budget on 3.14. Work handed to `asyncio.to_thread()` cannot be cancelled, so a thread still running at exit can hold the process well past any sane grace period — you will be `SIGKILL`ed in the middle of the join and never reach `loop.close()`. Either keep executor work short, or track it and refuse to enter shutdown while a long one is outstanding. ## State is the real constraint The decisive question is how much unflushed state you hold. A webhook receiver that accumulates a 2.4 GB working set of decoded events before writing cannot flush that inside a 30-second window on any realistic storage path, and no amount of shutdown engineering changes the arithmetic. The design fix is upstream of shutdown: checkpoint continuously so the amount at risk is always small, or do not accept the event until it is durable. That choice determines what you can promise. If handlers are idempotent and the sender retries, dropping in-flight work at the deadline is safe — at-least-once delivery with deduplication on a stable key. If they are not idempotent, every hard stop is a correctness event, and the honest response is to fix the handler rather than to lengthen the drain. ## The second signal Operators press Ctrl-C twice and supervisors escalate. The second signal should exit promptly rather than politely: stop draining, cancel immediately, exit. Because cleanup can be interrupted between two awaits, every shutdown path should be crash-safe anyway — write-then-rename, commit after durability, no half-updated structures across an `await`. If your process is only correct when it exits cleanly, it is not correct, because the machine can lose power. ## Verify it, don't assume it Shutdown is the least-tested path in most services and the one most likely to be exercised on every deploy. Make it a test: start the service under load, send `SIGTERM`, assert the exit code, assert the drain finished inside the budget, assert no partial records were written and that everything accepted was either completed or safely retryable. Export the drain duration as a metric — when it creeps toward the grace period you learn it from a dashboard rather than from a `SIGKILL` at 3am. ## The tradeoff to name out loud A longer drain means fewer interrupted requests and slower deploys, rollbacks and autoscaling; a shorter drain means fast, predictable restarts and more reliance on retries. Which one you pick is a product decision about the promise made to callers, not a tuning constant — and it belongs written down beside the grace period it must fit inside.