A service stops serving requests, the process is alive, and CPU is near zero. Working only from stack snapshots of every thread, how do you confirm it is a deadlock and identify which threads form the cycle?

## Start by classifying the hang "Requests stopped, process alive" has several possible causes, and the diagnosis path branches immediately on two cheap observations: **CPU usage** and **whether stacks change over time**. - CPU near zero, stacks frozen: threads are blocked. Candidates are deadlock or an unbounded external wait. - CPU high, stacks moving, no work completing: livelock or a retry storm. - CPU high, stacks in one hot region: a spin or an infinite loop. - Everything stopped periodically and then resumes: a pause of the runtime, not a lock problem. That is why the first action is to take **multiple snapshots** of all threads, spaced a few seconds apart. One snapshot cannot distinguish a thread that is stuck from a thread that is merely slow at that instant; three snapshots showing byte-identical frames for the same threads is strong evidence of a permanent block. Capture them *before* restarting, because a restart destroys the only evidence and the bug is timing dependent, so it may not recur for weeks. ## Extract the edges A snapshot of a blocked thread usually tells you two things: the call it is parked in and, if the runtime tracks ownership, which lock it is trying to acquire plus which locks it already owns. Those two facts are exactly the endpoints of a wait-for edge: *this thread* -> *the owner of the thing it wants*. Write one edge per blocked thread, then search for a cycle. A cycle whose members each hold what the next one is waiting for is a proof of deadlock, not a hypothesis. Many runtimes will do this for you and print an explicit "found a deadlock" section covering their built-in lock types. Treat that as a fast path, not as the definition: it only sees locks the runtime itself manages. ## The blind spot: resources the runtime does not know about Detectors are silent for anything the application implements itself or borrows from a library that keeps its own bookkeeping: - **Connection or object pool permits.** Every thread sits in "borrow from pool" and no owner is recorded. The dump looks like contention, not deadlock. - **Semaphore permits and bounded-queue slots.** A blocked put or take names no owner. - **Worker threads in a bounded pool** occupied by tasks that are waiting for other tasks that are still queued. - **Locks held remotely** - a database row lock, a distributed lease. For these you reconstruct the graph by reasoning about roles, not by reading ownership fields: group blocked threads by what they were doing, count how many hold each scarce resource, and ask whether the holders can ever finish. A useful tell is *N threads blocked acquiring a resource type whose total capacity is N and whose holders are themselves blocked* - a saturated pool whose holders are waiting is the classic resource deadlock, and no lock detector will ever report it. ## Rule out the lookalikes deliberately Before concluding deadlock, check three things. **Is there a cycle at all?** Frozen stacks parked on a socket read, an unbounded blocking receive, or a remote call with no timeout mean threads are waiting on the outside world. There is no cycle, so it is a hang: the fix is a timeout and a bounded wait, not lock ordering. **Do stacks really not change?** If frames advance slowly, you have a slow dependency or a convoy, not a deadlock. **Is one thread blocked while holding a lock across I/O?** This is not itself a deadlock but is the most common precursor: a thread parked in a network call while owning a lock stalls everyone behind it for as long as the call takes, and if the remote side is in turn waiting on this service, you have a genuine distributed cycle. ## Turn the finding into a fix A confirmed cycle names the exact resources and the exact acquisition orders involved, and that is the actionable output. Record the two orders observed - which thread took X then Y, and which took Y then X - because the remedy is chosen from that evidence. Also record whether the resources are runtime locks or application resources, since only the former are visible to future automatic detection; the latter needs metrics such as pool wait time, saturation, and borrow depth to be visible at all next time.