In a Go goroutine dump, how do you find which goroutine holds the sync.Mutex the others are waiting on?

## Why the dump refuses to help you directly A `sync.Mutex` is deliberately tiny: a state word and a semaphore word. It stores no owner, no recursion count and no acquisition site, which is part of why it is fast and why it is not reentrant. The consequence for diagnosis is blunt: a Go goroutine dump contains no line naming the holder of a lock. Engineers arriving from runtimes whose thread dumps print an explicit owner look for that line, do not find it, and conclude the dump is useless. It is not — the information is there, just as an inference. ## What the dump does give you Two things carry the signal. **The header line.** Each goroutine is introduced with its number, its wait reason and, once it has been blocked for more than a minute, its wait age: something of the form `goroutine 57 [sync.Mutex.Lock, 61 minutes]`. Current releases print a specific wait reason for mutex acquisition; older ones printed the more generic semaphore wait. The age is the part people underuse — it is a free timeline. **The frame arguments.** After each function name the runtime prints the first few words of that frame's arguments in hex. For a method the first word is the receiver pointer. So `sync.(*Mutex).lockSlow(0xc0000b4010)` tells you the address of the mutex being acquired, and `main.(*reconciler).sync(0xc0000b4000)` tells you which instance of your own type is running. Values the runtime is unsure about — registers that may have been reused at that program counter — are printed with a trailing `?`; treat those as hints rather than facts. ## The procedure 1. **Collect every goroutine whose stack contains a mutex acquisition** — frames under `sync.(*Mutex).Lock`, or `sync.(*RWMutex).Lock` and `RLock` if the lock is a read-write one. 2. **Read the receiver hex for that frame and group by it.** All goroutines sharing an address are queued on the same mutex instance. If you see two clusters, you have two contended locks and probably a cycle rather than simple contention. 3. **Find the holder.** It is the goroutine that has *already returned* from `Lock` on that address: its stack shows a function you know takes that mutex, with frames *above* it that are doing something else. It is parked on some other wait — a channel receive, a network read, a second `Lock`. If the mutex is a field of a struct, its address is the struct's address plus the field offset, so `main.(*reconciler).sync(0xc0000b4000)` sitting on a mutex at `0xc0000b4010` is very likely the same object; when the mutex is the first field, the two addresses are identical. 4. **Follow the second wait.** If the holder is blocked acquiring another mutex whose address is the one a third goroutine holds, and that one is blocked on the first, you have a cycle and you can name every participant. If instead the holder is blocked on a channel receive or a remote call, you have a chain, not a cycle — and the fix is usually a timeout or a context at the far end rather than lock reordering. 5. **Use the ages.** Everyone in a cycle shares roughly the same age. A cluster of waiters that are all younger than one goroutine points at that older goroutine as the head of the chain: it blocked first, and everyone else piled up behind it. ## Confirming that it is stuck and not slow Take a second dump a minute later. Same goroutine numbers, same frames, ages advanced by exactly the elapsed time means nothing moved. If the goroutine numbers change and the frames differ, you are looking at slow progress or contention, not a deadlock, and the investigation goes elsewhere. ## Traps worth knowing - **Inlining merges frames.** A small function that took the lock may not appear as its own frame; the acquisition looks like it happened in its caller. - **`sync.RWMutex` inverts the shape.** One pending writer blocks all later readers, so the visible symptom is a long queue of `RLock` waiters while the actual culprit is a single writer waiting behind an existing reader that will not finish. - **A deferred `Unlock` proves nothing about liveness.** `defer mu.Unlock()` guarantees release when the function returns; if the function never returns, the deferred call never runs and the lock is held forever. - **The dump is a snapshot with skew.** It is taken while the world is not perfectly still, so a single frame that looks odd is not necessarily meaningful; the pattern across goroutines is. ## When inference is not enough If a lock cycle keeps recurring and the stacks are ambiguous, stop guessing and make ownership explicit for one release: wrap the mutex in a small debug type that records the acquiring goroutine's identity and acquisition site behind a build flag, then remove it once the cycle is understood. That is a heavier tool than reading a dump, but it turns an inference into a fact — and, unlike the dump, it survives inlining.