An email-digest sender gets SIGTERM with a 30-second window before SIGKILL — how do you drain without truncating a digest?

## The contract you are being asked to implement A supervisor stopping a process sends a catchable request first and an uncatchable one later. The process gets a **grace window** — here 30 seconds — between the two. `signal.SIGKILL` and `signal.SIGSTOP` cannot be caught, handled, blocked or ignored; `signal.signal` raises `OSError` if you try. So the design question is never "how do I survive the kill" but "what state is the world in when the kill lands". ## Phase one: stop taking work The handler's only job is to record intent: ```python import signal import threading stopping = threading.Event() signal.signal(signal.SIGTERM, lambda signum, frame: stopping.set()) ``` Handlers run at a bytecode boundary in the main thread while the rest of the program is mid-flight. Logging, locking or network calls inside one can deadlock against whatever the interrupted code was holding. Setting an already-set `threading.Event` is idempotent, which matters because a second terminate signal is a normal occurrence. The dispatcher then stops claiming new digests. This is the single highest-value step: the set of work you must finish stops growing the instant the signal lands, so the drain has a bounded amount to do. ## Phase two: drain against a deadline, not against hope ```python import time deadline = time.monotonic() + 25.0 while in_flight and time.monotonic() < deadline: finish_one(in_flight.pop()) ``` Use `time.monotonic()`, never `time.time()`: a wall-clock adjustment mid-drain would move your deadline. Budget strictly less than the window — 25 seconds inside 30 — because after the drain you still have to flush buffers, close connections and let the interpreter finalize, and because the supervisor's clock started before yours did. A thread pool needs the same discipline. `ThreadPoolExecutor.shutdown(wait=True, cancel_futures=True)` stops the queue from being drained into workers and waits for the running tasks, but it has no timeout parameter, so a task that never returns holds you past the window. Either keep the per-item work short enough to be bounded, or track the deadline yourself and stop waiting when it passes. ## Where the truncation actually comes from The silent truncation in a digest sender is almost never caused by the drain being too short. It is caused by **ordering**: a process that streams sections into an outgoing message as it renders them, or that marks a digest as sent before the transport acknowledged it, will happily deliver a message containing the first four sections of twelve and record the job as complete. Nobody sees an error; the recipient just gets less than they should have. The ordering that survives being killed anywhere: 1. Render the digest completely, in memory or to a staging location. 2. Hand the finished artefact to the transport in a single operation. 3. Only after the transport acknowledges it, record "sent" and advance the cursor. Interrupted before step 3, the digest is re-sent next run — at-least-once rather than half-delivered. That is a property of the pipeline, not of the shutdown code, and it is what makes abandoning work safe. Deduplicate on a per-digest idempotency key if a duplicate would be worse than a delay. ## asyncio wears the same shape ```python import asyncio import signal async def drain_in_flight(): await asyncio.sleep(0.1) async def main(): loop = asyncio.get_running_loop() stopping = asyncio.Event() loop.add_signal_handler(signal.SIGTERM, stopping.set) loop.call_later(0.1, stopping.set) await stopping.wait() async with asyncio.timeout(5): await drain_in_flight() print("drained before the deadline") asyncio.run(main()) ``` `loop.add_signal_handler` is the asyncio-correct registration: the callback is scheduled on the event loop rather than running at an arbitrary bytecode boundary, so it may safely touch loop objects. `asyncio.timeout` (3.11+) gives the drain its deadline. Cancelling the remaining tasks and awaiting them with `return_exceptions=True` keeps one failing cleanup from hiding the others. ## Children and connections If the sender spawns children, forward the signal to them and give them their own, smaller budget: `Popen.terminate()`, then `Popen.wait(timeout=...)`, then `Popen.kill()` if the timeout expires. Otherwise your children outlive you or get killed mid-write. Connections deserve an explicit close inside the budget. A transport that is abandoned rather than closed can leave the peer holding a half-open connection until its own timeout fires, which is a slow way to turn a clean deploy into elevated latency elsewhere. ## Prove it Test it the way production stops you, not the way your terminal does: start the service, send `signal.SIGTERM` from a test harness with `os.kill`, and assert on the exit status and on the state of the work store — nothing marked sent that was not acknowledged, nothing lost that was not retryable. Record how long shutdown took and how often the process was still alive when the window expired; an exit reported as killed by signal 9 is the signal that your budget is wrong.