You're told a service is 'hung.' How would you tell whether it's suffering from deadlock, livelock, or starvation?
answer
- Two instruments: CPU usage + repeated thread dumps (jstack/jcmd a few seconds apart)
- Deadlock = low CPU, identical dumps, JVM prints 'Found one Java-level deadlock'
- Livelock = dumps change, throughput zero, CPU often high
- Starvation = system progresses but one thread perpetually behind
- CPU zero → deadlock; churn, no progress → livelock
basics
~20 sCheck CPU usage and thread dumps. Near-zero CPU with threads frozen in a cycle on each other's locks = deadlock. Threads that keep changing state, usually at high CPU, with no progress = livelock. The system works but one specific thread never advances = starvation.
solid answer
~50 sCombine a CPU-usage observation with repeated thread dumps (jstack / jcmd Thread.print) taken a few seconds apart. Deadlock: CPU near zero; threads are BLOCKED on monitors (WAITING if the locks are ReentrantLocks), and successive dumps are identical showing a stable cycle — the JVM's dump even prints a 'Found one Java-level deadlock' section detecting the cycle automatically. Livelock: CPU is usually high but the application makes no forward progress; dumps differ each time (threads keep changing state, often spinning in tryLock/retry or yield loops) yet throughput stays zero. Starvation: the overall system is making progress, but one specific thread (or a low-priority group) is repeatedly stuck waiting on the same lock or never appears RUNNABLE across dumps, while others keep advancing. The decision tree is: low CPU + a stable lock cycle → deadlock; changing dumps + zero progress → livelock; partial progress + one thread perpetually behind → starvation.
go deeper
Knows to look at CPU and capture a thread dump, and that deadlocked threads are BLOCKED (or WAITING on a Lock).
Uses jstack/jcmd, reads thread states, and recognizes the JVM's automatic deadlock section.
Applies the CPU + repeated-dump decision tree to separate all three failures, knows the JVM only auto-detects lock cycles, and reaches for JFR/profilers for contention.
Builds observability so these are caught proactively (lock-contention metrics, per-thread CPU, progress/heartbeat signals, alerts), and standardizes runbooks/diagnostics across services.
## The problem 'The service is hung' is a symptom, not a diagnosis. Three different liveness failures can all look like a hang from the outside, but they have **distinct fingerprints**. Telling them apart quickly saves a lot of time, because the fixes differ. ## Your two main instruments 1. **CPU usage** (top / OS metrics / per-thread CPU). This single number splits the cases fast: *is anything actually running?* 2. **Thread dumps.** A **thread dump** is a snapshot of each platform thread's state and stack (at JDK 27 `Thread.print` adds only the virtual threads mounted at that moment; `jcmd <pid> Thread.dump_to_file` lists virtual threads too). Capture it with `jstack <pid>`, `jcmd <pid> Thread.print`, or by sending the process a SIGQUIT. Each thread shows a **state** — notably `RUNNABLE` (running or ready), `BLOCKED` (waiting to enter a `synchronized` monitor), `WAITING`/`TIMED_WAITING` (parked, e.g. on a `Lock` or `Object.wait`). Take **two or three dumps several seconds apart** so you can see whether threads are *moving* between dumps. ## The fingerprints ### Deadlock — low CPU, stable, detectable - **CPU:** near zero. The threads are blocked, doing nothing. - **Dumps:** essentially **identical** across snapshots — the situation is frozen. Threads are `BLOCKED` on monitors (or `WAITING`, parked, on `ReentrantLock`s), each waiting for a lock another holds. - **Bonus:** the JVM **detects monitor deadlocks automatically** and prints a section like `Found one Java-level deadlock:` naming the threads and the cycle. (Note: it detects `synchronized`/`ReentrantLock` cycles, but not every kind of hang, e.g. a logical wait/notify lost-signal, nor any cycle that includes virtual threads.) ### Livelock — churning, usually high CPU, zero progress - **CPU:** usually **high** when the retry loop spins; lower if each retry waits or sleeps. - **Dumps:** **different each time** — threads keep changing state, often seen spinning in `tryLock`/retry loops, `Thread.yield()`, or condition rechecks. They're mostly `RUNNABLE` (or `TIMED_WAITING` inside a timed `tryLock` or back-off sleep), not parked at the same spot across dumps. - **Key tell:** business metrics (queue depth, processed count) **don't move** while the threads churn. Busy + no progress = livelock. ### Starvation — partial progress, one laggard - **CPU:** the *system* may be fine/busy; the **victim** thread shows little CPU. - **Progress:** the application **is** doing work overall — it's not fully hung — but **one specific thread or group never advances**. Across successive dumps that thread is perpetually `WAITING`/`BLOCKED` on the same lock, or rarely appears `RUNNABLE`, while peers keep moving. - **Common settings:** unfair lock with constant barging; where the OS honors Java priorities, a low-priority thread on a busy box; with virtual threads, CPU-bound virtual threads holding every carrier. ## The decision tree 1. **Is CPU near zero?** - Yes → look for `Found one Java-level deadlock` / a stable cycle of BLOCKED or lock-parked threads → **deadlock** (or a lost wait/notify if no cycle is reported). If the dumps keep changing instead, go to 2. - No → go to 2. 2. **Is throughput zero while the dumps change each time (CPU usually high)?** → **livelock**. 3. **Is the system making *some* progress while one thread/group is perpetually behind?** → **starvation**. ## Tooling beyond jstack - **VisualVM / JFR (Java Flight Recorder)** to watch per-thread CPU and lock contention over time. - **Async-profiler / lock profilers** to see which monitor is hot and who's barging. - For deadlock specifically, `jcmd <pid> Thread.print` includes the same deadlock-detection section as jstack. ## How to derive the answer Anchor on two questions: *Is anything running (CPU)?* and *Is the whole system stuck or just one thread (progress)?* Low CPU + frozen → deadlock; changing dumps + stuck → livelock; partial progress + one laggard → starvation. Two dumps a few seconds apart resolve almost every case.
- Why take multiple thread dumps a few seconds apart instead of one?A single snapshot can't tell you whether threads are stuck or merely passing through a state. Comparing dumps shows movement: identical dumps imply a frozen deadlock; constantly changing states with no progress imply livelock; one thread stuck across dumps while others move implies starvation.
- Does the JVM's automatic deadlock detection catch a lost wait/notify hang?Not necessarily. Its 'Found one Java-level deadlock' detection finds cycles of threads waiting on monitors/locks. A thread blocked in Object.wait() because a notify was missed has no lock cycle, so it won't be reported as a deadlock — you must diagnose it from the stacks.
- Assuming the JVM auto-detects all three — it only reports monitor/lock deadlock cycles
- Diagnosing from a single thread dump (you need two+ to see whether threads are moving)
- Equating high CPU with 'healthy' — high CPU plus zero progress is livelock
- Forgetting to check whether the system is making partial progress, which distinguishes starvation
Assumes JDK 27, where jstack and jcmd Thread.print show platform threads and only the mounted virtual threads, their deadlock search skips virtual threads, and jcmd Thread.dump_to_file lists virtual threads too.
context
- Thread.State (Java SE 27 & JDK 27)docs.oracle.com
- The jcmd Commanddocs.oracle.com