skip to content

Thread-Dump Patterns

Reading a series of thread dumps as patterns: a pool starving itself, a convoy behind one lock owner, threads parked on a connection pool, or virtual threads stuck on their carriers.

part ofJVM internalsread it →
on this pageshow

questions

4

In a HotSpot JVM thread dump, how do you find the one thread that a crowd of BLOCKED threads is queued behind?

level: juniorimportance: must knowfreq 55%

answer

  1. the hexadecimal address is the join key
  2. waiting to lock versus locked
  3. parking line names no owner
  4. Thread.print -l adds ownable synchronizers

basics

~20 s

Match the object address. Each BLOCKED thread prints waiting to lock <0x…>; the owner is the one thread whose stack prints locked <0x…> with the same address. Then read the owner's stack to see why it is not letting go.

solid answer

~50 s

For a `synchronized` monitor, each blocked thread's top frame carries `- waiting to lock <0x…> (a com.example.StockCache)`, and the owner is the one thread whose stack prints `- locked <0x…>` with the same address, under the frame that entered the block. If that owner is itself `waiting to lock` another address, follow it: that is the chain, and if it closes on itself among platform threads, `jcmd <pid> Thread.print` reports it at the end under `Found one Java-level deadlock:`. A `ReentrantLock` is not a monitor: waiters print `- parking to wait for <0x…> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)` and nobody prints `locked`, so take the dump with `jcmd <pid> Thread.print -l` and find the address under some thread's `Locked ownable synchronizers:`. Finally read the owner's own frames across two or three dumps: an owner sitting in slow I/O is a convoy, not a deadlock.

code

console · 12 lines
console
"http-worker-7" #41 [18233] daemon prio=5 ...
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.example.inventory.StockCache.reserve(StockCache.java:88)
	- waiting to lock <0x00000006c3a0e1b8> (a com.example.inventory.StockCache)
	at com.example.orders.OrderService.place(OrderService.java:142)

"http-worker-3" #37 [18229] daemon prio=5 ...
   java.lang.Thread.State: RUNNABLE
	at com.example.pricing.PriceClient.fetch(PriceClient.java:57)
	at com.example.inventory.StockCache.reload(StockCache.java:131)
	- locked <0x00000006c3a0e1b8> (a com.example.inventory.StockCache)
	at com.example.orders.OrderService.place(OrderService.java:139)

go deeper

for a junior

Recall the two lines that pair up: waiting to lock on the waiter and locked on the owner, joined by the same address.

for a middle

Explain why a ReentrantLock owner never prints locked, and that Thread.print -l adds the Locked ownable synchronizers section that names it.

for a senior

Show you can tell a convoy from a chain from a cycle using a series of dumps, and name the owner's slow frame as the lead.

for a principal

Talk about making lock owners easy to find: named threads, few long-held locks, and dumps that list ownable synchronizers by default in your runbook.

## What a HotSpot thread dump prints about locks A **thread dump** taken with `jcmd <pid> Thread.print` (the same output `jstack` asks for) prints every platform thread's stack and, under each frame, the lock lines HotSpot knows about. Four lines matter for finding who is holding everyone up: - `- locked <0x…> (a com.example.StockCache)`: this frame entered a `synchronized` block or method on that object and still holds its **monitor**. - `- waiting to lock <0x…> (a …)`: printed under the top frame of a thread that is BLOCKED trying to enter that monitor. - `- waiting on <0x…> (a …)`: printed under `Object.wait()`. The thread gave the monitor up and waits for a notify, so it is **not** an owner. - `- parking to wait for <0x…> (a …)`: the thread is parked in a `java.util.concurrent` lock, condition or similar; HotSpot's classic print names the blocker object, never its owner. The hexadecimal value is the object's address when the dump was taken, and inside one dump it is the **join key**: the same object prints the same address in every thread. Between two dumps a collector may have moved the object, so across a series match by class and code location rather than by address. ## Following the chain, step by step 1. Group the BLOCKED threads by the address in their `waiting to lock` line. Forty threads on one address is one hot lock; forty threads spread over forty addresses is not a queue behind one owner. 2. Search the same dump for `locked <that address>`. Normally one thread prints it: the **owner**. 3. Read the owner's top frames. If it is RUNNABLE in application code or inside a remote call, the chain ends there and that frame is your lead. 4. If the owner is itself `waiting to lock` a second address, repeat step 2 with that address. That is a **lock-owner chain**. 5. If the chain comes back to a thread already on it, it is a cycle. `jcmd <pid> Thread.print` runs HotSpot's deadlock finder after the stacks and, for a cycle among platform threads, prints it under `Found one Java-level deadlock:`. ## java.util.concurrent locks need -l A `ReentrantLock` is not a monitor, so no frame of its owner prints `locked`. Two ways to find the owner: - Re-take the dump with `jcmd <pid> Thread.print -l`. Every thread then ends with a `Locked ownable synchronizers:` list (or `- None`), and the owner is the thread listing the address the waiters park on. Without `-l` that section is not printed at all. - On JDK 25, `jcmd <pid> Thread.dump_to_file <file>` captures the owner directly: its plain format appends `, owner #<tid>` to the parking line, and its JSON format writes an `owner` inside `parkBlocker`. Only an **exclusively owned** synchronizer has an owner thread. A condition wait, a semaphore permit or a latch has none, and for those the question changes from "who holds it" to "who is supposed to signal it". ## Convoy, chain or cycle | Pattern | What one dump shows | What a series of dumps shows | |---|---|---| | **Convoy** behind one owner | Many waiters on one address; the owner RUNNABLE in slow work, often I/O, while holding it | The owner may change, but there is always one thread inside the same slow section and the queue never drains | | **Lock-owner chain** | The owner of A is waiting to lock B; the owner of B is working | Stable while B's owner works, then clears | | **Deadlock** | The chain closes on itself | Identical in every dump; reported as a Java-level deadlock when only platform threads and owned locks are involved | | Passing contention | A short queue, different addresses | Different threads each time, queue drains | A convoy is the pattern people most often misread as a deadlock. Nothing is stuck forever: every request needs the same lock, and the code holding it does something slow, such as a `synchronized` cache refresh that makes a network call. Throughput collapses to one request at a time. The evidence that names it is the owner's stack showing the slow call **below** a frame that prints `locked`. The fix comes after that evidence: move the slow work out of the critical section, which is a lock-design question rather than a triage one. ## Traps when reading the lines - `jcmd <pid> Thread.print` prints at most `MaxJavaStackTraceDepth` frames per thread (1024 by default), so on a very deep stack the frame that took the lock can fall past the cut. - A lock the JIT compiler removed prints as `eliminated`, not `locked`. - A monitor held by a virtual thread that is not mounted on a carrier has no owner in the classic print, and HotSpot's deadlock finder skips virtual threads entirely; use `Thread.dump_to_file` for those. - One dump cannot tell a convoy from a deadlock. Take three, a few seconds apart, and compare the owner's frames.

  • Why does an owner sometimes show up with no matching waiting to lock line anywhere?
    Because `locked` only means the frame holds the monitor; nobody has to be waiting for it. Most owners in a healthy dump hold uncontended locks. The line only matters when other threads print `waiting to lock` with the same address, so always start from the waiters and work towards the owner, not the other way round.
  • Three dumps show the same forty waiters, but the owning thread is different in each. Deadlock or convoy?
    A convoy. A deadlock freezes the same threads on the same locks in every dump. Here the lock keeps changing hands, so work is moving, just one request at a time, and each owner is presumably inside the same slow section while holding it. Read the owners' frames below the `locked` line to find that section.
  • Why can the address of the contended lock change between two dumps?
    The printed value is the object's address at that instant, and a moving collector can relocate the object between dumps. Within one dump it is a reliable join key; across dumps match the lock by its class and the code location that takes it instead.

  • A thread printing waiting on <0x…> is the owner the others are blocked behind.
  • Forty BLOCKED threads always mean a deadlock, so restart and look no further.
  • In jcmd Thread.print output, the parking to wait for line names the ReentrantLock's owner.
  • Thread.print -e adds the java.util.concurrent lock owners to the dump.
  • One thread dump is enough to tell a convoy from a deadlock.
open as a page

An order service on a HotSpot JVM stops answering after a traffic burst with CPU near idle, and three thread dumps show every HTTP worker waiting in a JDBC connection-pool borrow: what do you look for next?

level: seniorimportance: must knowfreq 60%

basics

~20 s

Look past the waiters to whoever holds the connections. Holders stuck in a driver socket read point at the database; holders waiting on something else, such as a second borrow, point at your code; too few holders means connections leaked.

open as a page

On a JDK 25 service running virtual threads, why take jcmd Thread.dump_to_file -format=json rather than Thread.print, and how does a pinned carrier show up?

level: middleimportance: should knowfreq 45%

basics

~20 s

Thread.print shows a virtual thread only while it is mounted on a carrier, so parked ones are missing. Thread.dump_to_file lists all of them by container; in its JSON, a WAITING virtual thread that still has a carrier is pinned.

open as a page

A HotSpot thread dump shows every worker of a small fixed thread pool parked in FutureTask.get, and Thread.print reports no deadlock: what is happening, and how do you prove it?

level: seniorimportance: should knowfreq 50%

basics

~20 s

Thread-starvation deadlock: each worker runs a task waiting on futures of subtasks queued in the same pool, and no worker is free to run them. HotSpot reports nothing because a FutureTask has no owner thread, so there is no lock cycle.

open as a page