schedtrace and Stack Dumps

## The signal and what the runtime does with it SIGQUIT is a POSIX signal whose default operating-system behaviour is to terminate the process and write a core file. The Go runtime installs its own handler for it, and that handler is why the signal matters here: **before the process dies, the runtime walks every live goroutine and prints its state and stack to standard error.** There are three ordinary ways to send it: - press `Ctrl-\` in the terminal running the program in the foreground; - `kill -QUIT <pid>` from another shell (`kill -3 <pid>` is the same signal by number); - have a supervisor or orchestrator send it to the process. Nothing has to be prepared in advance. There is no build tag, no `-gcflags`, no imported package and no HTTP endpoint involved — any Go binary, including one you did not compile, responds this way unless it has deliberately taken the signal over. ## What the output looks like The dump is a sequence of blocks, one per goroutine, each introduced by a header line: ``` goroutine 1 [select]: goroutine 4711 [chan receive, 143 minutes]: ``` The header carries three things: the runtime's goroutine id, the goroutine's state — and, when it is blocked, the *reason* it is blocked, such as `chan receive`, `chan send`, `select`, `sleep`, `IO wait` or `semacquire` — and, when it has been waiting for a minute or longer, how long it has been waiting, rounded down to whole minutes. The runtime records the moment a goroutine parked, so that duration is real elapsed blocked time, not CPU time. Those headers are usually where the answer is. In a process with tens of thousands of goroutines you do not read the dump linearly; you scan the headers for repeated states and for large ages. A thousand goroutines all reading `[chan receive, 200+ minutes]` is a different story from a thousand reading `[IO wait]` with no age at all. ## The cost: it is fatal This is the half candidates forget. After printing, the runtime restores the default disposition for SIGQUIT and re-raises it, so the process is *killed by the signal*, not exited cleanly. Concretely: - deferred functions do not run; - no shutdown hooks, flushes or connection drains happen; - every in-flight request on that instance is lost; - you get **one** dump from that process, ever. So SIGQUIT is not something you sprinkle over a fleet. You choose an instance you can afford to lose — a canary, a replica behind a load balancer that will be restarted anyway, a job that will be retried — and you take the dump deliberately, knowing it is the last thing that instance does. ## What can stop it working - **The program claimed the signal.** If it calls `signal.Notify(ch, syscall.SIGQUIT)` — or `signal.Ignore` — the runtime hands SIGQUIT to the program's own channel and its default dump-and-die path never runs. Daemons that treat SIGQUIT as "finish current work and stop" do exactly this, and in exchange they give up the runtime's best free diagnostic. - **Nothing forwards the signal.** A shell wrapper, an init process or a container runtime may absorb `Ctrl-\` or send the signal to a PID 1 shim rather than to your binary. - **stderr goes nowhere.** The dump is written to file descriptor 2. If the process was started with stderr pointing at `/dev/null`, or your logging setup only captures stdout, the dump is produced and thrown away. Check where stderr lands *before* you fire the signal, because you do not get a second try. - How much detail the traceback contains is further governed by the `GOTRACEBACK` setting, which is the knob to reach for when the default output is not enough — or is too much. ## Where it fits among the other runtime diagnostics Think of the three cheap, no-rebuild sources as answering different questions. `runtime.NumGoroutine()` answers *how many goroutines exist* — a number, over time, from inside the program. `GODEBUG=schedtrace=1000` answers *what is the scheduler doing* — how many Ps are idle, how deep the run queues are — printed periodically while the process keeps running. The SIGQUIT dump answers *what is each individual goroutine waiting on, and for how long* — once, at the cost of the process. Used in that order, they compose: the counter tells you something is wrong, schedtrace tells you whether the goroutines are competing for CPU or parked, and the dump tells you precisely what they are parked on.