Requests handled by a worker pool have stopped completing and throughput has collapsed. How do you determine whether the pool is merely saturated by slow work or genuinely deadlocked, and why does the answer change what you do about it?

## Three failure shapes that look the same from outside From a dashboard, a stalled pool, a saturated pool, and a livelocked pool all read as 'requests time out'. They need different responses, so distinguishing them is the whole job. - **Saturation:** arrivals exceed service capacity, or service time grew. Workers are busy doing real work or blocked on a real dependency. Progress continues at a reduced rate. - **Deadlock:** a set of workers waits on conditions only they could satisfy - most often parents blocked on results from their own pool, or two tasks acquiring two locks in opposite orders. Progress is exactly zero and permanent. - **Livelock or thrash:** threads are running (CPU is busy) but useful completions are near zero - retry loops, contention collapse, spinning on a contended structure. Distinguished from deadlock by CPU being high rather than idle. ## The diagnostic ladder **1. Completion counter over a window.** The single most informative signal: tasks completed per second. Non-zero and steady, however small, means the system is alive - saturated or dependency-slow. Exactly zero across a minute while work is queued means stuck. **2. Two stack samples.** Capture the state of all worker threads twice, several seconds apart, and diff them. Same task identity plus same frame equals no progress on that worker. All workers frozen equals deadlock. Some frozen, some rotating equals partial starvation - often a subset of workers is stuck while the rest still serve. **3. Read the frames.** Where a worker is parked names the cause: waiting for a result of a submitted task means the nested-submission pattern; waiting on a lock means a lock-ordering cycle; waiting in a socket read means a slow dependency with no read timeout, which is saturation dressed up as a hang. **4. Queue depth trend.** Growing with zero completions equals deadlock. Growing with non-zero completions equals overload. Flat and non-empty with zero completions and idle CPU is the strongest deadlock signature. **5. CPU.** Deadlocked pools burn no CPU; livelocked ones burn plenty. This one number separates 'everyone is waiting' from 'everyone is spinning'. **6. Load experiment, if you can afford it.** Cut arrivals to near zero. Saturation drains and recovers; deadlock stays exactly where it was. This is the definitive test, because the defining property of a deadlock is that it is not load-dependent once entered. ## Why the distinction dictates the response For **saturation** the levers are load and capacity: shed or throttle arrivals, enforce deadlines so abandoned work stops occupying workers, cap concurrency to the slow dependency so it cannot consume the whole pool, add workers if the bottleneck is genuinely wait-bound, or reduce service time. The system heals itself once the imbalance is corrected, and no restart is needed. For **deadlock** none of those help. Adding workers postpones recurrence; shedding load does not release the held threads; the process must be restarted to recover, and until the structure changes it will happen again at the next burst. The fix is structural: remove same-pool blocking, isolate pools by dependency layer, impose a global lock order, or make waiting non-blocking. Timeouts on waits are worth adding as a safety net - they convert a permanent hang into a bounded failure and let the pool recover on its own - but a timeout is containment, not a cure. ## Making it observable in advance Instrument pools with active worker count, queue depth, completed count, and task age of the oldest in-flight task. A rising 'oldest in-flight task age' with a flat completed count is the earliest machine-readable signature of a stuck pool, and it is the alert worth having, because by the time latency alerts fire the pool is already unrecoverable.