skip to content

Production Troubleshooting

What to do when a live JVM misbehaves: split a slowdown into GC, safepoint, CPU, lock and native-memory causes, each with its own evidence. Senior rounds open with 'the p99 doubled'.

part ofJVM internalsread it →
on this pageshow

explore

questions

page 1 of 2

In a HotSpot JVM thread dump, how do you find the one thread that a crowd of BLOCKED threads is queued behind?

level: juniorimportance: must knowfreq 55%

answer

  1. the hexadecimal address is the join key
  2. waiting to lock versus locked
  3. parking line names no owner
  4. Thread.print -l adds ownable synchronizers

basics

~20 s

Match the object address. Each BLOCKED thread prints waiting to lock <0x…>; the owner is the one thread whose stack prints locked <0x…> with the same address. Then read the owner's stack to see why it is not letting go.

solid answer

~50 s

For a `synchronized` monitor, each blocked thread's top frame carries `- waiting to lock <0x…> (a com.example.StockCache)`, and the owner is the one thread whose stack prints `- locked <0x…>` with the same address, under the frame that entered the block. If that owner is itself `waiting to lock` another address, follow it: that is the chain, and if it closes on itself among platform threads, `jcmd <pid> Thread.print` reports it at the end under `Found one Java-level deadlock:`. A `ReentrantLock` is not a monitor: waiters print `- parking to wait for <0x…> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)` and nobody prints `locked`, so take the dump with `jcmd <pid> Thread.print -l` and find the address under some thread's `Locked ownable synchronizers:`. Finally read the owner's own frames across two or three dumps: an owner sitting in slow I/O is a convoy, not a deadlock.

code

console · 12 lines
console
"http-worker-7" #41 [18233] daemon prio=5 ...
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.example.inventory.StockCache.reserve(StockCache.java:88)
	- waiting to lock <0x00000006c3a0e1b8> (a com.example.inventory.StockCache)
	at com.example.orders.OrderService.place(OrderService.java:142)

"http-worker-3" #37 [18229] daemon prio=5 ...
   java.lang.Thread.State: RUNNABLE
	at com.example.pricing.PriceClient.fetch(PriceClient.java:57)
	at com.example.inventory.StockCache.reload(StockCache.java:131)
	- locked <0x00000006c3a0e1b8> (a com.example.inventory.StockCache)
	at com.example.orders.OrderService.place(OrderService.java:139)

go deeper

for a junior

Recall the two lines that pair up: waiting to lock on the waiter and locked on the owner, joined by the same address.

for a middle

Explain why a ReentrantLock owner never prints locked, and that Thread.print -l adds the Locked ownable synchronizers section that names it.

for a senior

Show you can tell a convoy from a chain from a cycle using a series of dumps, and name the owner's slow frame as the lead.

for a principal

Talk about making lock owners easy to find: named threads, few long-held locks, and dumps that list ownable synchronizers by default in your runbook.

## What a HotSpot thread dump prints about locks A **thread dump** taken with `jcmd <pid> Thread.print` (the same output `jstack` asks for) prints every platform thread's stack and, under each frame, the lock lines HotSpot knows about. Four lines matter for finding who is holding everyone up: - `- locked <0x…> (a com.example.StockCache)`: this frame entered a `synchronized` block or method on that object and still holds its **monitor**. - `- waiting to lock <0x…> (a …)`: printed under the top frame of a thread that is BLOCKED trying to enter that monitor. - `- waiting on <0x…> (a …)`: printed under `Object.wait()`. The thread gave the monitor up and waits for a notify, so it is **not** an owner. - `- parking to wait for <0x…> (a …)`: the thread is parked in a `java.util.concurrent` lock, condition or similar; HotSpot's classic print names the blocker object, never its owner. The hexadecimal value is the object's address when the dump was taken, and inside one dump it is the **join key**: the same object prints the same address in every thread. Between two dumps a collector may have moved the object, so across a series match by class and code location rather than by address. ## Following the chain, step by step 1. Group the BLOCKED threads by the address in their `waiting to lock` line. Forty threads on one address is one hot lock; forty threads spread over forty addresses is not a queue behind one owner. 2. Search the same dump for `locked <that address>`. Normally one thread prints it: the **owner**. 3. Read the owner's top frames. If it is RUNNABLE in application code or inside a remote call, the chain ends there and that frame is your lead. 4. If the owner is itself `waiting to lock` a second address, repeat step 2 with that address. That is a **lock-owner chain**. 5. If the chain comes back to a thread already on it, it is a cycle. `jcmd <pid> Thread.print` runs HotSpot's deadlock finder after the stacks and, for a cycle among platform threads, prints it under `Found one Java-level deadlock:`. ## java.util.concurrent locks need -l A `ReentrantLock` is not a monitor, so no frame of its owner prints `locked`. Two ways to find the owner: - Re-take the dump with `jcmd <pid> Thread.print -l`. Every thread then ends with a `Locked ownable synchronizers:` list (or `- None`), and the owner is the thread listing the address the waiters park on. Without `-l` that section is not printed at all. - On JDK 25, `jcmd <pid> Thread.dump_to_file <file>` captures the owner directly: its plain format appends `, owner #<tid>` to the parking line, and its JSON format writes an `owner` inside `parkBlocker`. Only an **exclusively owned** synchronizer has an owner thread. A condition wait, a semaphore permit or a latch has none, and for those the question changes from "who holds it" to "who is supposed to signal it". ## Convoy, chain or cycle | Pattern | What one dump shows | What a series of dumps shows | |---|---|---| | **Convoy** behind one owner | Many waiters on one address; the owner RUNNABLE in slow work, often I/O, while holding it | The owner may change, but there is always one thread inside the same slow section and the queue never drains | | **Lock-owner chain** | The owner of A is waiting to lock B; the owner of B is working | Stable while B's owner works, then clears | | **Deadlock** | The chain closes on itself | Identical in every dump; reported as a Java-level deadlock when only platform threads and owned locks are involved | | Passing contention | A short queue, different addresses | Different threads each time, queue drains | A convoy is the pattern people most often misread as a deadlock. Nothing is stuck forever: every request needs the same lock, and the code holding it does something slow, such as a `synchronized` cache refresh that makes a network call. Throughput collapses to one request at a time. The evidence that names it is the owner's stack showing the slow call **below** a frame that prints `locked`. The fix comes after that evidence: move the slow work out of the critical section, which is a lock-design question rather than a triage one. ## Traps when reading the lines - `jcmd <pid> Thread.print` prints at most `MaxJavaStackTraceDepth` frames per thread (1024 by default), so on a very deep stack the frame that took the lock can fall past the cut. - A lock the JIT compiler removed prints as `eliminated`, not `locked`. - A monitor held by a virtual thread that is not mounted on a carrier has no owner in the classic print, and HotSpot's deadlock finder skips virtual threads entirely; use `Thread.dump_to_file` for those. - One dump cannot tell a convoy from a deadlock. Take three, a few seconds apart, and compare the owner's frames.

  • Why does an owner sometimes show up with no matching waiting to lock line anywhere?
    Because `locked` only means the frame holds the monitor; nobody has to be waiting for it. Most owners in a healthy dump hold uncontended locks. The line only matters when other threads print `waiting to lock` with the same address, so always start from the waiters and work towards the owner, not the other way round.
  • Three dumps show the same forty waiters, but the owning thread is different in each. Deadlock or convoy?
    A convoy. A deadlock freezes the same threads on the same locks in every dump. Here the lock keeps changing hands, so work is moving, just one request at a time, and each owner is presumably inside the same slow section while holding it. Read the owners' frames below the `locked` line to find that section.
  • Why can the address of the contended lock change between two dumps?
    The printed value is the object's address at that instant, and a moving collector can relocate the object between dumps. Within one dump it is a reliable join key; across dumps match the lock by its class and the code location that takes it instead.

  • A thread printing waiting on <0x…> is the owner the others are blocked behind.
  • Forty BLOCKED threads always mean a deadlock, so restart and look no further.
  • In jcmd Thread.print output, the parking to wait for line names the ReentrantLock's owner.
  • Thread.print -e adds the java.util.concurrent lock owners to the dump.
  • One thread dump is enough to tell a convoy from a deadlock.
open as a page

In a HotSpot hs_err fatal error log, what do the signal line and the 'Problematic frame:' line tell you, including the frame's first letter?

level: middleimportance: must knowfreq 38%

basics

~20 s

The signal line gives the signal, its number and the faulting pc. The problematic frame's first letter says whose code ran: C native code outside the VM, V HotSpot's own C++, v VM-generated stubs, J JIT-compiled Java, j interpreted Java.

open as a page

In async-profiler, which event mode answers which question about a live JVM, and what does each mode's flame graph actually count?

level: middleimportance: must knowfreq 50%

basics

~20 s

Use cpu for where CPU time is burned, wall for elapsed thread time including waits, alloc for which call sites allocate heap, and lock for waits to enter contended locks; each graph counts only its own resource.

open as a page

A HotSpot JVM in a Kubernetes pod throws OutOfMemoryError and gets restarted, but no heap dump survives. How do you make that evidence outlive the restart?

level: middleimportance: must knowfreq 48%

basics

~10 s

Enable -XX:+HeapDumpOnOutOfMemoryError with -XX:HeapDumpPath pointing at a volume that outlives the container, add -XX:+ExitOnOutOfMemoryError so the JVM exits once the dump is written, and size the volume for a whole heap.

open as a page

A dashboard shows a JVM service's p99 latency doubled overnight; what do you verify about the measurement itself before investigating the JVM?

level: middleimportance: must knowfreq 60%

basics

~20 s

Confirm the regression is real before blaming the JVM: the same percentile from the same metric and histogram, enough requests behind it, the same traffic mix and load, and whether it shows on every instance or only some.

open as a page

On a HotSpot JVM running JDK 25, how do you use Native Memory Tracking to find which part of the JVM's native memory keeps growing?

level: middleimportance: must knowfreq 55%

basics

~10 s

Start the JVM with -XX:NativeMemoryTracking=summary, take jcmd <pid> VM.native_memory baseline once it is warm, then run summary.diff later: the category whose committed figure keeps rising names the growing subsystem.

open as a page

A pricing service's async-profiler CPU flame graph shows a wide regex plateau, while a wall-clock profile of the same minute shows request threads in a socket read: which explains slow requests, and why?

level: seniorimportance: must knowfreq 52%

basics

~20 s

The wall-clock profile explains the slow requests: latency is elapsed time, and threads blocked in a socket read use no CPU, so a CPU profile cannot see them. The regex plateau is the largest share of CPU, not of request time.

open as a page

A checkout API on a JVM went from about 120 ms to 260 ms p99 after a release, while p50 and errors stayed flat; how do you split the extra tail time by cause?

level: seniorimportance: must knowfreq 65%

basics

~20 s

Attribute the extra tail time to JVM stops (GC pauses and time-to-safepoint), on-CPU work, off-CPU waiting on locks, pools or I/O, or slower downstream calls, reading the always-on logs first and adding one targeted recording per surviving hypothesis.

open as a page

When a JVM in a 2 GiB container with a 1 GiB max heap gains resident memory for two days until OOM-killed while post-GC heap and all NMT categories stay flat, where is the memory going, and how do you prove it?

level: seniorimportance: must knowfreq 50%

basics

~20 s

Flat NMT with rising RSS points outside the JVM's own allocations: a native library leaking malloc'd memory, or glibc arenas retaining freed memory. VM.info's outstanding-versus-retained line, a native heap trim and async-profiler's nativemem leak profile tell them apart.

open as a page

An order service on a HotSpot JVM stops answering after a traffic burst with CPU near idle, and three thread dumps show every HTTP worker waiting in a JDBC connection-pool borrow: what do you look for next?

level: seniorimportance: must knowfreq 60%

basics

~20 s

Look past the waiters to whoever holds the connections. Holders stuck in a driver socket read point at the database; holders waiting on something else, such as a second borrow, point at your code; too few holders means connections leaked.

open as a page

On a HotSpot JVM, what is an hs_err_pid<pid>.log file, when is one written, and where do you look for it?

level: juniorimportance: should knowfreq 32%

basics

~20 s

An hs_err_pid<pid>.log file is the fatal error log HotSpot writes when the JVM process itself crashes, for example on a SIGSEGV in native code. It lands in the working directory by default, falls back to the temp directory, and -XX:ErrorFile moves it.

open as a page

In an async-profiler CPU flame graph of a JVM, how do you find where time is really spent, and what do plateaus and colours show?

level: middleimportance: should knowfreq 45%

basics

~20 s

Look for wide frames with nothing on top: a plateau is self time, while a wide frame with children only passes time to callees. Colours give the frame type: compiled, inlined or interpreted Java, native, JVM C++ or kernel.

open as a page

On Linux, how does jcmd attach to a running HotSpot JVM, and why does that handshake break when the tool and the JVM sit in different containers?

level: middleimportance: should knowfreq 38%

basics

~20 s

jcmd drops an .attach_pid trigger file, sends SIGQUIT, and waits for the JVM to open a .java_pid UNIX socket in /tmp. Across containers the PID, the /tmp directory and the user can all differ, and each mismatch breaks it.

open as a page

After changing a JVM service to fix its p99 regression, how do you prove with before-and-after evidence that the change worked?

level: middleimportance: should knowfreq 50%

basics

~20 s

Measure the same percentile from the same histogram at the same load and traffic mix, over a window long enough to contain the events behind the tail, ideally canary beside baseline, and check that the mechanism you blamed moved too.

open as a page

On a HotSpot JVM, why does Native Memory Tracking's committed total rarely match the process's resident set size, and which way can they differ?

level: middleimportance: should knowfreq 45%

basics

~20 s

NMT committed counts memory the JVM made usable; RSS counts pages actually in RAM. Committed exceeds RSS when committed pages are untouched; RSS exceeds committed when native code, C-library retention or file mappings hold memory NMT never tracked.

open as a page

In the safepoint log a JDK 25 HotSpot JVM prints with -Xlog:safepoint, what do the Reaching safepoint, At safepoint and Threads fields each tell you?

level: middleimportance: should knowfreq 40%

basics

~20 s

Each line names one VM operation and splits its stop into nanosecond phases: Reaching safepoint is the wait for every thread to stop, At safepoint is the operation's own work, and Threads counts the threads it had to wait for.

open as a page

On a JDK 25 service running virtual threads, why take jcmd Thread.dump_to_file -format=json rather than Thread.print, and how does a pinned carrier show up?

level: middleimportance: should knowfreq 45%

basics

~20 s

Thread.print shows a virtual thread only while it is mounted on a carrier, so parked ones are missing. Thread.dump_to_file lists all of them by container; in its JSON, a WAITING virtual thread that still has a carrier is pinned.

open as a page

A JVM in production died three times this week with exit codes 134, 137 and 3. How do you tell a HotSpot crash from a kernel OOM kill or an ExitOnOutOfMemoryError exit?

level: seniorimportance: should knowfreq 36%

basics

~20 s

134 is SIGABRT, HotSpot aborting after a crash, with an hs_err file; 137 is SIGKILL, typically the kernel OOM killer, leaving no file but a kernel log record; 3 is -XX:+ExitOnOutOfMemoryError exiting after a 'Terminating due to java.lang.OutOfMemoryError' line.

open as a page

A JVM service that calls a native image-processing library through JNI dies several times a week with exit code 134, leaving hs_err_pid<pid>.log files. How do you establish whose bug it is?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Compare the problematic frames and crashing threads across several hs_err files. A C frame in the library with the thread in _thread_in_native points at the library or the JNI glue; a compiler thread points at the JIT; V frames in GC threads need native corruption ruled out.

open as a page

When async-profiler -e cpu on a JVM inside a Docker container reports perf events unavailable, or shows no kernel frames, why does that happen, and what are your options?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Docker's default seccomp profile or a restrictive perf_event_paranoid blocks perf_event_open, so async-profiler falls back to ctimer, which cannot collect kernel stacks. Accept ctimer, force user-space perf_events, profile from the host as a privileged user, or relax seccomp deliberately.

open as a page

During triage of a HotSpot JVM in a Kubernetes pod, how do you check which CPU count and memory size the JVM believes it has, and how does JDK 25 derive them?

level: seniorimportance: should knowfreq 36%

basics

~20 s

Ask the JVM: the container section of jcmd <pid> VM.info shows its CPU count and limits. On JDK 25 the count is quota over period, rounded up and capped by usable CPUs; the memory limit stands in for physical memory.

open as a page

A Kubernetes pod's JVM image has no jcmd; from an ephemeral debug container with a full JDK, jcmd first finds no JVM, then times out once you target the app container. What is wrong, and how do you attach?

level: seniorimportance: should knowfreq 42%

basics

~20 s

Without a shared PID namespace the debug container cannot see the JVM. Sharing it fixes that, but a different user cannot reach the JVM's /tmp, so the trigger lands in the wrong one. Run as the JVM's UID and GID.

open as a page

A JVM service's p99 doubled after a release that also moved its pods to a new Kubernetes node pool; how do you find which change caused it?

level: seniorimportance: should knowfreq 50%

basics

~20 s

List every change, not only the code: dependencies, base image and JDK, flags, CPU limits, node hardware. Then separate them by experiment, such as the old build on the new nodes, and compare what the JVM chose at startup on each side.

open as a page

When a JDK 25 HotSpot service slows after days of uptime and its stderr shows "CodeHeap 'non-profiled nmethods' is full. Compiler has been disabled.", what happened, and how do you confirm and fix it?

level: seniorimportance: should knowfreq 35%

basics

~20 s

The JIT's code cache segment for optimised code filled, so HotSpot stopped compiling and new or recompiled hot methods run slower code. Confirm with jcmd Compiler.codecache and the CodeCacheFull event, then size the cache from max_used with headroom.

open as a page

Why can the same int-indexed hot loop stall time-to-safepoint for hundreds of milliseconds on JDK 25 under the Parallel collector but not under G1, and how do you fix it?

level: seniorimportance: should knowfreq 30%

basics

~20 s

C2 drops the safepoint poll from int-counted loops unless UseCountedLoopSafepoints is on; G1, ZGC and Shenandoah turn it on with LoopStripMiningIter 1000, while Serial and Parallel leave it off, so a long loop polls only at its exit.

open as a page

A JDK 25 HotSpot safepoint log shows Reaching safepoint near 400 ms with one runnable thread; how do you identify that late thread and the code it was running?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Name the thread with -XX:+SafepointTimeout and a SafepointTimeoutDelay below the stall, which logs the threads that had not arrived; then find its code with an async-profiler --ttsp profile, which samples only between the safepoint request and synchronization.

open as a page

On a JDK 25 HotSpot service using the Parallel collector, stopped time exceeds GC pause time and -Xlog:safepoint names operations that are not collections; how do you find who stopped the application?

level: seniorimportance: should knowfreq 36%

basics

~20 s

Group the safepoint log by operation name, keep the non-collection names, map each to its requester (PrintThreads to a thread dump, GC_HeapInspection to a class histogram, RedefineClasses to an agent), then correlate timestamps to find the caller.

open as a page

A HotSpot thread dump shows every worker of a small fixed thread pool parked in FutureTask.get, and Thread.print reports no deadlock: what is happening, and how do you prove it?

level: seniorimportance: should knowfreq 50%

basics

~20 s

Thread-starvation deadlock: each worker runs a task waiting on futures of subtasks queued in the same pool, and no worker is free to run them. HotSpot reports nothing because a FutureTask has no owner thread, so there is no lock cycle.

open as a page

As the lead for a fleet of JVM services, would you run async-profiler continuously in production, and if so in which modes, at what interval and overhead budget?

level: principalimportance: should knowfreq 35%

basics

~20 s

Usually yes, narrowly: CPU, and perhaps wall-clock, allocation and lock at coarse intervals, as rotating JFR files with the agent loaded at startup. Set the overhead budget up front and measure it; broad or fine-grained modes stay on demand.

open as a page

As the lead for a fleet of JVM services on Kubernetes, what diagnosability would you require every JVM image and pod spec to carry, and what would you trade against hardening?

level: principalimportance: should knowfreq 30%

basics

~20 s

Keep every JVM attachable (writable /tmp, a known UID, a matching JDK debug image), make failure evidence automatic (heap dump on OOM to a volume, exit after it, a continuous recording), and decide deliberately which attach powers to switch off.

open as a page

showing 1–30 of 34