In a GODEBUG=schedtrace=1000 line, what do `runqueue=12` and the trailing `[3 0 2 0]` mean?

## What the line is Setting `GODEBUG=schedtrace=1000` in the environment makes the Go runtime emit **one line to standard error every 1000 milliseconds** summarising the state of the goroutine scheduler. It requires no rebuild, no imports and no code change — it is a runtime knob read at process start-up, so it works on a binary somebody else built. A line looks roughly like this: ``` SCHED 5008ms: gomaxprocs=4 idleprocs=1 threads=9 spinningthreads=0 needspinning=0 idlethreads=4 runqueue=12 [3 0 2 0] ``` `SCHED 5008ms` is time since process start, so the lines are self-timestamping relative to start-up. ## The fields - **`gomaxprocs`** — the number of Ps, the scheduling contexts. This is the ceiling on how many goroutines can be executing Go code at the same instant. - **`idleprocs`** — how many of those Ps have no goroutine to run right now. `idleprocs` equal to `gomaxprocs` means the program, at this instant, has nothing runnable at all. - **`threads`** — the total OS threads (Ms) the runtime has created. It is normally a little above `gomaxprocs`, since threads are also spent on blocking syscalls and on the runtime's own monitor. - **`spinningthreads`** / **`needspinning`** — threads actively looking for work, and whether the scheduler wants another one to start looking. Persistently non-zero spinning means the scheduler keeps waking up to find nothing. - **`idlethreads`** — threads parked with no P. - **`runqueue`** — the number of runnable goroutines sitting in the single **global** run queue. - **`[3 0 2 0]`** — one number per P, the number of runnable goroutines in that P's **local** run queue. The list is always `gomaxprocs` entries long, so it is easy to see skew between Ps. ## The single most important property: it counts runnable work only Every queue number on the line is about goroutines that *could run if a P were free*. A goroutine parked on a channel receive, waiting on a mutex, sleeping, or blocked in the network poller is not in any run queue and is invisible here. **The summary line never reports how many goroutines exist.** A process with 40,000 goroutines all blocked and a process with 3 goroutines all blocked produce indistinguishable lines. That sounds like a limitation, and it is exactly what makes the line useful: it cleanly separates *too much runnable work* from *work that is stuck*. ## Reading the common shapes - **`idleprocs=0` and `runqueue` climbing** — every P is busy and the backlog is growing. The program is offering more runnable goroutines than it has Ps to run them on: CPU-bound, or under-provisioned with respect to `GOMAXPROCS`. Check what `GOMAXPROCS` actually resolved to for this process before concluding the code is slow. - **`idleprocs` equal to `gomaxprocs`, all queues zero** — nothing is runnable. Either the process is genuinely idle, or its goroutines are all blocked. If a goroutine count is simultaneously climbing, this is the signature of accumulation-by-blocking rather than of a scheduling problem. - **Local queues badly skewed, e.g. `[97 0 0 0]`** — one P is holding a deep local queue while others idle, usually a transient before work stealing evens it out; persistent skew is worth a closer look. - **`threads` far above `gomaxprocs`** — threads are being spent on something other than running Go code, typically blocking syscalls or cgo calls. ## Detail mode Adding `scheddetail=1` — the whole setting becomes `GODEBUG=schedtrace=1000,scheddetail=1` — replaces the one-line summary with a multi-line dump at the same cadence: a line per P (its status, its schedule and syscall tick counters, the M it is attached to, its local queue size), a line per M, and **a line per goroutine** giving a numeric status and, for a waiting goroutine, its wait reason in parentheses. There are no stacks, but you finally get what the summary omits: how many goroutines exist and what each one is doing. Detail mode is much more expensive. It locks the scheduler while it walks every P, M and G, and it prints one line per goroutine *per interval* — with tens of thousands of goroutines that is megabytes of stderr a minute and a measurable slowdown. Run it on one instance, at a slow interval, for as long as you need and no longer. ## Don't build tooling on it The runtime documents this output as unspecified and free to change; fields have come and gone between releases and the detail format especially is an internal debugging aid. Use schedtrace interactively, during an investigation. For anything that must keep working across upgrades — dashboards, alerts, capacity reports — read the runtime's supported metrics interface instead.