Given a ThreadPoolExecutor whose latency is rising, how do you combine its monitoring signals (activeCount, poolSize, queue size, completedTaskCount) to diagnose whether the pool is saturated, mis-sized, or stalled?

## The goal: turn vital signs into a diagnosis A pool exposes a handful of numbers; latency rising tells you *something* is wrong but not *what*. The skill is reading the **combination** to distinguish three very different conditions, because the fix for each is opposite. ## The four signals (recap) - **`getActiveCount()`** — threads running a task right now (approximate). - **`getPoolSize()` / `getMaximumPoolSize()`** — threads that exist vs the configured ceiling. - **`getQueue().size()`** — tasks waiting (backlog). - **`getCompletedTaskCount()`** — cumulative finished tasks; sampled over time it gives the **completion rate** (throughput). ## Pattern 1: genuine saturation (too much work) **Signature:** `activeCount` approx `maximumPoolSize` (all workers busy), `queue.size()` **trending up**, and the completion rate is **steady but maxed out**. The pool is doing all it can; arrival rate > service rate, so work piles up and latency (queue wait) climbs. **Fix:** add capacity (more threads if the work is I/O-bound, more machines), shed/throttle load, or apply backpressure. Raising the queue does *not* help — it just hides latency. ## Pattern 2: stall / deadlock (work is stuck) **Signature:** `activeCount` is **high** (threads look busy) but the **completion rate collapses** — `getCompletedTaskCount()` barely increases over successive samples — and the queue may grow too. The threads are *occupied but not progressing*: blocked on slow I/O, contending on a lock, or, classically, waiting on a result that must be produced by **another task on the same pool that can never get a thread** (pool-induced deadlock). **Fix:** this is the dangerous case because the saturation signature looks similar, but **adding threads barely helps and sizing isn't the root cause**. You must find what blocks the workers (thread dump shows them parked/blocked), and remove the blocking dependency — e.g. never have a pooled task block waiting on another task submitted to the *same* pool; isolate blocking work on a separate pool. ## Pattern 3: under-utilization (pool isn't the bottleneck) **Signature:** `queue.size()` near 0, `activeCount` well **below** `poolSize`, completion rate healthy. The pool keeps up easily; rising latency is **upstream or downstream** (network, DB, a single-threaded stage feeding it). Don't touch the pool. ## Pattern 4: mis-sized for the workload shape `getLargestPoolSize() == getMaximumPoolSize()` tells you the ceiling was reached. If you sized the pool with a CPU-bound rule (e.g. ~#cores) but tasks actually **block on I/O**, the right thread count is much higher (cores / (1 - blocking fraction)). The fix is re-sizing for the real blocking profile or moving to async/non-blocking I/O. ## The decisive measurement: completion-rate trend The single most discriminating signal is **delta `getCompletedTaskCount()` over time**: - Busy threads + **steady** completion rate + growing queue = **saturation** (load). - Busy threads + **collapsing** completion rate = **stall** (blocked work) — a thread dump confirms. - Idle threads + empty queue = **not the pool**. Always **sample over time** and correlate the **queue slope** with the **completion-rate slope**; a single snapshot can't tell saturation from a stall. ## Practical instrumentation Poll the four accessors on a scheduled monitor (every few seconds), compute the queue slope and completion rate, and alert on (a) sustained queue growth, (b) completion-rate collapse while active, and (c) largestPoolSize hitting max. Pair with thread dumps for the stall case. The per-task `beforeExecute`/`afterExecute` hooks add per-task latency histograms that confirm whether individual tasks got slower (stall) or just waited longer in queue (saturation).