Why does a containerized Python process exit with status 137 and print no traceback?

## What the number actually is Unix does not have an exit code for "killed by a signal"; it has a wait status that says either *exited with code N* or *terminated by signal N*. Shells and container runtimes flatten those two cases into one number by reporting a signalled death as **128 + signal**. SIGKILL is 9 on Linux, so you see 137. Nothing in your program chose that number -- there is no `sys.exit(137)` anywhere. From inside Python the same event looks different: - `subprocess.run` reports a negative return code, -9, and - if you decode a raw wait status yourself, `os.WIFSIGNALED` is true and `os.WTERMSIG` is 9. ## Who did the killing When a container runs under a memory limit, the kernel's **cgroup memory controller** charges every page the process faults in against that limit. When a charge cannot be satisfied and reclaim (dropping page cache, swapping if allowed) does not free enough, the kernel invokes the **OOM killer** scoped to that cgroup and sends SIGKILL to a process inside it. Selection, timing and delivery are all kernel-side; the interpreter is a bystander that is simply stopped between two bytecode instructions. If the process killed is the container's PID 1, the whole container ends. ## Why Python prints nothing CPython's signal handling is deliberately **cooperative**. The C-level handler does almost nothing except set a flag; the actual Python callback runs later, in the eval loop, between bytecodes. SIGKILL never reaches that machinery at all, because POSIX forbids catching, blocking or ignoring it -- `signal.signal(signal.SIGKILL, handler)` raises `OSError`. The consequences are total: - no `MemoryError` is raised, - no `except` block runs, - no `finally` runs, - no context manager's `__exit__` runs, - no `atexit` callback fires, - no `__del__` runs, and - the logging module never gets its shutdown flush. `faulthandler` does not rescue you either: it exists to dump a traceback on fatal faults such as SIGSEGV or SIGABRT, and SIGKILL is not one of the signals it can arm. ## The missing last log line When stdout is a pipe rather than a terminal it is **block-buffered**, so several kilobytes of already-written log lines can be sitting in the buffer when the kill lands. The logs then point at an earlier place in the code than where the process actually died, which sends people hunting the wrong function. Running with `PYTHONUNBUFFERED=1` or `python -u`, or using a logging handler that flushes each record, makes the trail honest -- at some throughput cost. ## 137 is not a synonym for out-of-memory Any SIGKILL from any sender produces it. The most common non-OOM source is a **shutdown sequence**: the platform sends SIGTERM, waits a grace period, and escalates to SIGKILL when the process has not exited. A Python service that ignores SIGTERM, or that is blocked in a long call that never checks for the signal, will show 137 on every single deploy and nothing is wrong with its memory at all. Tell the two apart with evidence rather than assumption: - the cgroup's `memory.events` file carries an `oom_kill` counter that increments only for real OOM kills, - the kernel ring buffer records a "Killed process" line, and - the container runtime records an OOM-killed exit reason. A clean SIGTERM death, by contrast, shows 143 (128 + 15) and *is* catchable -- Python can install a SIGTERM handler and shut down gracefully. ## Reading the evidence from each vantage point The same death looks different depending on who is watching. - A shell or an orchestrator's status field shows 137. - A Python parent that spawned the process sees a return code of -9 from `subprocess.run`, or `os.WIFSIGNALED` true with `os.WTERMSIG` returning 9 if it decodes a raw wait status itself. - The kernel log shows the cgroup that ran out, the chosen victim's process id and command, and its **resident size at the moment of death** -- which is often the single most useful number in the whole incident, because it tells you what the process was actually holding rather than what it was averaging. Knowing all three views matters because in most deployments you can only reach one of them, and a supervisor that logs only "child exited with -9" is reporting the same event as a dashboard that says OOMKilled. ## What follows for design Because there is no in-process hook, everything useful must be **pre-emptive**. - Log or export the *peak* memory rather than the average, since the kill is driven by an instantaneous high-water mark that a 30-second metrics scrape will usually miss. - Cap concurrency so peak footprint is bounded by construction. - And treat sudden death as a design constraint rather than an exceptional case: work items must be **idempotent and redeliverable**, because any cleanup you were planning to do in a `finally` block is not going to happen.