Why do you take multiple thread dumps over time, and how do you read them to diagnose stuck threads and contention?

## The core limitation: a dump is one instant A thread dump is a **snapshot** — it freezes the JVM's threads at a single microsecond. That instant cannot tell you about *duration*. A thread shown inside `processOrder()` might be: - **stuck there forever** (the bug), or - **passing through** it normally (perfectly healthy). The snapshot looks identical in both cases. **Duration is invisible in one dump.** This is the fundamental reason single dumps mislead. ## The fix: sample over time Taking **multiple dumps spaced a few seconds apart** turns the snapshot into a crude **time series**. A common recipe: 3–5 dumps, ~5 seconds between each (sometimes scripted in a loop). Now duration becomes observable: | Observation across dumps | Interpretation | |---|---| | Same thread, **same stack frame** in every dump | **Stuck** — it has not moved; investigate that frame | | Same thread, **different frames** each dump | Healthy — it is doing work and progressing | | A monitor with a **growing count of BLOCKED** threads | **Contention** building on that lock | | Threads steadily **accumulating in WAITING** on a pool | **Resource exhaustion** (connections/threads running out) | The key insight: you're inferring *time-in-state* from *repeated samples*, the same way a sampling profiler infers hot methods from periodic stack samples. ## Reading for a stuck thread Pick a suspect thread (often one whose stack sits in your code or an external call) and trace its stack across the dumps. If frames `a()->b()->c()` appear unchanged in dump 1, 2, and 3, that thread has been parked on `c()` the whole window. Common culprits: an external call with no timeout (HTTP/DB), an infinite loop, a wait on a condition that never signals. If the top frame is a blocking I/O call (e.g. `socketRead0`), the thread is waiting on a slow/dead downstream — note that such a thread still shows **RUNNABLE** (the native-I/O caveat). ## Reading for lock contention Contention is a **distribution** phenomenon, best seen across dumps: 1. In each dump, group threads by the monitor they are `waiting to lock`. 2. If the same monitor has many waiters **persistently** across dumps, that critical section is a bottleneck. 3. Identify the **owner** — the one thread that `locked` that monitor — and look at *its* stack: it's doing the slow work (e.g. a long computation or I/O while holding the lock) that serializes everyone else. 4. The fix follows from the owner's stack: shrink the critical section, move I/O out of the lock, use a finer-grained or read/write lock, or a lock-free structure. ## Practical capture Script it so the dumps are timestamped and aligned: ``` for i in 1 2 3 4 5; do jstack $PID > dump_$i.txt # or: jcmd $PID Thread.print sleep 5 done ``` Then diff or load the set into an analyzer (e.g. fastthread.io, IBM TDA) which clusters identical stacks and highlights threads that never moved and monitors with many waiters. For an intermittent production hang you may also trigger capture automatically on a latency/CPU alarm. ## Why not just one big profiler run? A sampling profiler gives smoother data, but jstack needs **nothing installed**, has negligible overhead per dump, and works headless over SSH — ideal for a live incident. The multi-dump technique is the cheapest way to add the missing *time* dimension to a snapshot tool.