Popen.terminate() sometimes fails to stop a translation-memory updater's export child; how would you build a shutdown that always ends it?

"Terminate then kill" is one of the few genuinely universal patterns in process management — it is what init systems, container runtimes and orchestrators all do — and an interviewer asking this wants the whole state machine, including the parts people leave out. ## The scenario, concretely A translation-memory updater runs nightly for an 11-person team. For each locale it spawns an export child that streams entries out to a file, formatting dates and numbers in that locale's own conventions. When the run is cancelled, the current code calls `Popen.terminate()` and moves on. Two failure modes follow. Some children are still alive afterwards, because SIGTERM reached a wrapper and not the exporter, or because the exporter was inside a call that delayed delivery. Others were killed midway through writing a record, and because the output format is locale-dependent there is no unambiguous end-of-record marker — a truncated line looks like a shorter but valid one, so the next run merges corrupt entries instead of failing loudly. Both failures come from treating shutdown as a single call. ## Stage 0: make the target signalable, at spawn time You cannot fix this at shutdown time. Start the child with `start_new_session=True`, which calls `os.setsid()` in the child so it leads a new session and process group and every helper it forks inherits that group. Cache the group id right away with `os.getpgid(p.pid)` — after `Popen.wait()` reaps the child the pid is gone and may be recycled, so a late `os.getpgid()` either raises `ProcessLookupError` or, worse, resolves against an unrelated process. ## Stage 1: ask, with a deadline `os.killpg(pgid, signal.SIGTERM)` reaches the exporter and anything it started. Then `p.wait(timeout=grace)`. The grace period is a real design parameter, not a magic number: it should be a little longer than the child's worst realistic time to finish the record it is on and close the file, and shorter than whatever deadline you are operating under — the job's own timeout, the deployment's shutdown budget, the orchestrator's grace period before it SIGKILLs *you*. Being killed yourself mid-escalation is the failure people forget, so your budget has to fit inside your supervisor's. For this workload the child should also make the grace period useful: writing to a temp file and renaming it into place on success means an interrupted export leaves no half-file at all, which is a stronger guarantee than any grace period alone. ## Stage 2: escalate, and wait again On `subprocess.TimeoutExpired`, `os.killpg(pgid, signal.SIGKILL)` and then `p.wait()` with no timeout. The second wait is not decoration. `os.killpg()` returning only means the signal was queued; the child is not gone until the kernel has torn it down and you have reaped it. Skipping that wait is how you get a shutdown routine that reports success while the process still holds the output file. Both `killpg` calls can raise `ProcessLookupError` — the group emptied between your check and your signal — and that is a normal, ignorable outcome, not an error to propagate. Guard the whole sequence with an early `p.poll() is not None` return so you never signal a group id whose members have all exited. ## Stage 3: make it observable and reentrant Log which stage ended each child: exited on its own, exited after SIGTERM, or required SIGKILL. That single field turns an invisible problem into a trend — if the SIGKILL share climbs after a release, some child stopped honouring SIGTERM, and you will know before the corrupt output shows up in the memory. Make the routine idempotent, because it will be called concurrently from a signal handler and from the normal cleanup path, and both may fire during a real shutdown. ## Propagating your own shutdown The parent must also handle being asked to stop. Install `signal.signal(signal.SIGTERM, handler)` in the parent and have the handler run the same escalation for every live child before exiting. Do not rely on `atexit` alone: it runs on a normal interpreter exit but not when the parent itself is SIGKILLed, and in that case nothing of yours runs at all. That is the argument for the process group being a real one — an external supervisor, a systemd unit or a container runtime can then clean up the whole group even when your parent dies without warning. One trap: `with subprocess.Popen(...) as p:` does not kill anything. `__exit__` closes the pipes and then calls `wait()`, so a child that never exits turns the context manager into an indefinite hang. If you want the block to bound the child's life, the escalation belongs in a `finally`. ## What the sequence buys you SIGTERM first gives well-behaved children their cleanup path; the deadline stops a hung child holding the pipeline; SIGKILL guarantees termination; the group makes both signals reach the whole tree; the second wait makes the outcome true rather than assumed. Drop any one of the five and you have one of the two failures the team started with.