An order service on a HotSpot JVM stops answering after a traffic burst with CPU near idle, and three thread dumps show every HTTP worker waiting in a JDBC connection-pool borrow: what do you look for next?
answer
- the waiters are only the symptom
- find the threads holding connections
- a held connection leaves no lock line
- driver socket read, other wait, or nobody
basics
~20 sLook past the waiters to whoever holds the connections. Holders stuck in a driver socket read point at the database; holders waiting on something else, such as a second borrow, point at your code; too few holders means connections leaked.
solid answer
~50 sThe borrowers parked inside the pool are the symptom; they typically park on a condition or semaphore inside the pool, which has no owner thread, so the dump will not name a culprit for them. The evidence is in the threads that **hold** the pool's connections. A held connection prints no lock line, so find holders by their frames: threads inside the JDBC driver, or threads below a transaction-boundary frame in your code. Then split three ways. Holders RUNNABLE in a driver socket read on the same statement in all three dumps mean the database is slow or blocked, so look at the database's own session and lock views next. Holders parked on something else while holding a connection, a remote call, a lock or a second borrow from the same pool, mean hold-and-wait in your code. Far fewer holders than the pool's maximum size means connections were leaked. Only then change one thing.
go deeper
Recall that threads waiting for a connection are victims, and that the useful question is what the threads holding connections are doing.
Explain why a condition wait has no owner in the dump, and how to recognise holders by driver frames and transaction-boundary frames.
Classify holders into database-slow, hold-and-wait, self-deadlock or leak from three dumps, and name the evidence each points to before touching pool size.
Argue for borrow timeouts, leak detection and a rule against remote calls inside transactions, so the next stall announces its cause.
## The waiters are not the evidence A **connection pool** lends a fixed number of database connections to request threads. When none is free, the borrower waits inside the pool's code. In a HotSpot thread dump those borrowers all look alike: WAITING or TIMED_WAITING, the top frames inside the pool's borrow method, and a line such as `- parking to wait for <0x…> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)`. A condition wait or a semaphore permit has **no owner thread**, so no `locked` line and no `owner` will lead you anywhere from here. Two quick readings are still worth taking: - `TIMED_WAITING` means the borrow has a timeout and requests will eventually fail with an error; plain `WAITING` means a borrow that can wait forever, which is a finding in itself. - The **count** of waiters against the pool's maximum size tells you how deep the queue is, not why it formed. ## Find the holders The question that matters is what the threads that **hold** the connections are doing. A connection is just an object reference on someone's stack, so holding one prints nothing. Find holders by their frames instead: 1. Threads with the JDBC driver's frames on their stack are using a connection right now. 2. Threads whose stack runs through a transaction-boundary frame in your code (the method that opened the transaction) hold one even if their top frames are somewhere else entirely. 3. Compare the number of holders you found with the pool's maximum size, and compare each holder's frames across the three dumps taken five seconds apart. ## Three patterns, three different next steps | Holders look like | It means | Next evidence | |---|---|---| | RUNNABLE in a driver socket read, same statement in every dump | The database is slow, or the statements are waiting on database-side row locks | The database's session, wait and lock views for those statements | | Holding a connection but parked elsewhere: a remote HTTP call, a lock, a future | Hold-and-wait in the application: a slow non-database step inside the transaction | The frame they are parked in, and that dependency's own latency | | Holding one connection and parked in a **second borrow** from the same pool | Pool self-deadlock: every connection is held by a thread waiting for another | The code path that borrows twice, often a nested transaction | | Far fewer holders than the maximum size | Connections borrowed and never returned: a **leak** | The pool's leak detection or borrow-stack logging | Two details make the first row easy to misread. A thread blocked in a socket read inside native code is reported as RUNNABLE, because from the JVM's point of view it is executing a native method; it is not burning CPU, which matches the idle processors. And the same statement in all three dumps rules out "just busy": the same query is not still running ten seconds later on a healthy database. The self-deadlock row is the one that matches a burst exactly. Under light load a request that holds one connection and borrows a second always finds one free. After a burst, every connection may be held by a request that is waiting for its second, and nothing is ever returned. HotSpot's deadlock finder will not report it: the borrowers wait on the pool's condition or permit, which has no owner thread, so there is no cycle for it to follow. ## Rule things out before changing anything - **Not CPU**: idle processors and RUNNABLE-in-native holders agree with each other. - **Not the JVM's lock detector**: no `Found one Java-level deadlock:` does not mean no deadlock, as the self-deadlock row shows. - **Not the pool size by default**: raising the maximum helps only if the holders are healthy and the queue is simply deeper than the pool. If the database is the bottleneck, more connections mean more concurrent statements competing for the same locks, and the stall gets worse. If the code borrows twice, a bigger pool only moves the burst size at which it deadlocks again. ## Order of work 1. Confirm the series: the same thread ids in the same frames across all three dumps means stuck, not slow. 2. Classify the holders into one of the rows above. 3. Collect the evidence that row names. 4. Change one thing, such as moving the remote call out of the transaction or removing the second borrow, and compare a fresh series of dumps under the same load.
- Why do the threads holding connections in a socket read show as RUNNABLE while CPU is idle?Thread state is the JVM's view: a thread executing a native method, including one blocked in the kernel on a socket read, is RUNNABLE. It consumes no CPU while it waits. So RUNNABLE in a driver read frame plus idle processors means waiting on the database, not working.
- How would you confirm a pool self-deadlock rather than a leak?In a self-deadlock every connection has a visible holder, and each holder is itself parked in the pool's borrow method under a frame that already opened a transaction. In a leak, the holders you can find are fewer than the pool's size, because the leaked connections belong to no live stack at all.
- The borrowers show TIMED_WAITING instead of WAITING. Does that change the diagnosis?Not the diagnosis, the consequence. A timed borrow throws after its timeout, so requests fail with errors and the service may recover once the burst passes, while an untimed one can hang until restart. Either way the cause is in what the holders are doing.
- Raise the maximum pool size first and see whether the waits go away.
- The borrowing threads are deadlocked, so Found one Java-level deadlock will name them.
- Holders in a socket read are RUNNABLE, so they must be burning CPU.
- A thread holding a connection always shows JDBC driver frames on its stack.
- No deadlock reported by Thread.print means the pool cannot be deadlocked.