A Python worker pool renders no more invoices — how do you read an all-threads stack dump to find the stuck frame?

## The method, before the details A wedged process has three plausible states and you can separate them with two dumps and one CPU reading: | CPU | Two dumps compared | Diagnosis | |---|---|---| | ~0% | identical | Threads are **blocked** — waiting on a lock, a condition, a socket, a child process | | one core pinned | identical *or* moving | A thread is **spinning** — a loop that never exits, or a tight C call | | busy, spread | moving | The process is **working**, just slower than you expected | The single most common mistake is taking one dump, seeing everything parked in `wait`, and announcing a deadlock. One sample cannot distinguish stopped from slow. Take the second. ## Reading a worker's frames A `faulthandler` all-threads dump prints, per thread, the id and (on 3.14) the `Thread.name`, then frames **innermost first**. So the top line is where the thread is *now*, and the lines below are how it got there. Common innermost signatures: - **`threading.py` in `wait`, under `queue.py` in `get`.** The worker is idle, parked on an empty work queue. This is a healthy worker in an unhealthy process: if all eight look like this, the pool is not stuck at all — whatever feeds it has stopped. Go and look for the feeder thread; if it is missing from `threading.enumerate()`, it died with an exception you never logged. - **`threading.py` in `wait` or in a lock acquisition, under your own code.** Genuine contention. Now find the thread that is *not* waiting, because it is the holder. - **A socket read, a subprocess wait, a file read.** Blocked on the outside world. The dump tells you *which* external dependency; the fix is a timeout, not a lock change. - **The same application frame on every worker, with the process burning CPU.** Not a lock at all: a hot loop, or a data-dependent path that got quadratic. ## The subtle case: identical stacks at 100% CPU This looks like a contradiction and it is the case worth being able to explain. The dump prints a **line number**, not a bytecode offset. A thread grinding through a tight loop whose body spans two lines can produce byte-identical dumps seconds apart while consuming a full core. So never conclude "blocked" from identical stacks alone — pair the dumps with the CPU reading. Equally, a thread inside a single long-running C call will show one unchanging Python frame no matter how much work it is doing, because the dump cannot see into the extension. ## Working the invoice-renderer example A renderer's pool of eight workers stops completing jobs. The dumps show seven workers parked in a lock acquisition under `render_document`, and one worker whose innermost frame is a `while` line in `paginate` — the same line in both dumps, with one core pinned. That combination reads cleanly: the eighth worker entered a pagination loop whose exit condition is off by one at the final page boundary, so it never leaves; it holds the lock the other seven want; the pool's throughput drops to zero without a single error being logged. The 2.4 GB working set is a corroborating detail rather than the cause — it stops growing when the pool stops, which tells you the process is not still allocating, it is stalled. Notice what the dump did **not** tell you: it did not say the loop is wrong, only that the process is inside it and staying there. The dump narrows a whole service to one function; you still read that function. ## Cross-checks that make the reading solid - **`threading.enumerate()` alongside the dump.** Confirms the expected worker count and gives you `ident`-to-name mapping. A worker missing from both is a worker that died. - **Per-thread CPU from the operating system**, joined on `Thread.native_id`, tells you exactly which thread is the spinner rather than making you infer it. - **A third dump.** If a stack is unchanged across three samples spread over a minute, "it is just slow" stops being a credible story. - **Resident memory over the same interval.** Flat memory with flat stacks is a stall; climbing memory with flat stacks means something is still accumulating on your behalf. ## Limits to state out loud The dump covers Python frames in this interpreter only. Native frames are invisible — a thread deep inside a compiled extension shows only the Python frame that called in, so if every dump points at the same call into an extension, the next tool is an operating-system-level debugger, not another dump. And a thread that never touched Python at all will not appear with frames worth reading.