A crashed queue worker's GOTRACEBACK=all dump shows 300 goroutines parked in `chan receive` for 40 minutes — how do you read it?

## The dump is not a list of suspects A `GOTRACEBACK=all` crash dump prints every goroutine in the process. In a queue-consuming worker that is mostly a wall of goroutines parked in `chan receive` — which is exactly what an idle worker pool looks like. The reading job is to separate the one goroutine that failed from the several hundred that are merely waiting, and then to turn the waiting ones into a statement about the code. ## Step 1 — find the crash, not the crowd The failure is at the top of the output: the `panic:` or `fatal error:` line, followed by the goroutine that produced it, which is in state `running` (for a deadlock detected by the runtime, there is no running goroutine at all — that is itself the finding). Everything after that first goroutine is context. Two misreadings to avoid: `goroutine 1` is the goroutine running `main`, not necessarily the one that crashed; and the goroutine printed last has no special meaning. ## Step 2 — collapse identical stacks Three hundred goroutines with the same frames are one fact, not three hundred. Group them by stack signature — the ordered list of function names, ignoring the hex words and the goroutine ids. What survives is usually a handful of distinct groups: the workers, an HTTP or gRPC-shaped listener, a metrics ticker, the runtime's own housekeeping. Now each group is one question. ## Step 3 — read the `created by` line Each non-main goroutine's stack ends with ``` created by main.(*Consumer).start in goroutine 1 /srv/worker/consume.go:66 +0x11d ``` which is the `go` statement that made it, by file and line, plus the id of the parent goroutine. That line is what takes you from "a group of stuck goroutines" to a specific loop in a specific function — often the only link, because nothing in the stack itself mentions the code that set the work up. If all three hundred share one `created by` line and near-consecutive ids, they were spawned by one loop; if their ids are spread across the whole range, they accumulated over the process's lifetime, which is a very different story. ## Step 4 — ask who was supposed to be on the other end A goroutine parked in `chan receive` is waiting for a sender. So search the dump for one. Read every group's frames and ask which goroutine owns the send side of that channel. Three outcomes: - **There is a sender, also blocked** — you have a cycle; read its state and its channel too. - **There is a sender, running or in `IO wait`** — the pipeline is alive and slow, not stuck; the duration is your evidence for which. - **There is no sender anywhere** — the producer returned, and because the channel was never closed the receivers will wait forever. This is the common shape behind "the worker went quiet". The fix is on the producer's exit path, not on the workers. The duration field turns this from a guess into a claim: forty minutes on a queue that should deliver continuously says the sender has been gone for forty minutes, and the ids and `created by` lines say when and by whom the receivers were created. ## Step 5 — know what the dump is hiding At the default level the runtime elides its own frames, so a parked goroutine's top frame is your function, not the `runtime.gopark`/`runtime.chanrecv` pair underneath it. Re-reading the crash with the traceback level raised to `system` restores those frames and adds the runtime's own goroutines, which tells you precisely which runtime primitive each goroutine parked in — useful when your top frame is ambiguous, noisy otherwise. Deep stacks are truncated with `...additional frames elided...`. A goroutine executing on another thread may print `goroutine running on other thread; stack unavailable` instead of frames. And a dump is one instant: it shows where goroutines *are*, never where they have been, so a fast pipeline and a stopped one can look similar for anything under a minute of waiting. ## What raises the answer above a procedure The judgment an interviewer is listening for is that **blocking is not a bug**. The finding is never "goroutines are blocked on a channel"; it is "these workers, created by this line, have been waiting this long, and no goroutine in this process is capable of sending to them." Everything above is how you get to that sentence from a wall of text. One modern aid is worth knowing: in Go 1.27 tracebacks carry a goroutine's `runtime/pprof` labels for modules that declare `go 1.27` or later, so labels attached with `pprof.Do` — a queue name, a tenant, a message kind — appear beside the stack. In a dump of three hundred otherwise identical workers, that is often the difference between "they are all stuck" and "every one of them is stuck on the same queue".