Runaway OS Threads

## Two different populations A Go program has two kinds of "concurrent thing" and they are counted by two different bookkeepers. A **goroutine** is the runtime's own unit of concurrency: a function launched with `go f()`, with a small growable stack, scheduled entirely inside the Go runtime. Creating one costs a couple of kilobytes and no system call. An **OS thread** (the runtime calls it an **M**, for machine) is a real kernel thread created with a system call. The Go runtime multiplexes many goroutines onto a much smaller set of Ms, and it creates a new M when it needs one so that runnable goroutines can keep running. The two numbers are related only loosely, which is exactly why you need to read both. ## Counting goroutines ```go n := runtime.NumGoroutine() ``` This returns the number of goroutines that currently exist — running, runnable, and blocked. It is cheap, it is safe to call from anywhere, and it is the number `/debug/pprof/goroutine` also reports at the top of its output. It is **not** a thread count, and no amount of goroutine growth tells you directly what the kernel sees. ## Counting OS threads There is no `runtime.NumThreads()`. Four practical readings exist. **1. `runtime/metrics`.** Go 1.26 added scheduler gauges, and `/sched/threads:threads` is the current number of OS threads the runtime is managing. This is the reading to export to your metrics system, because it is cheap, live, and needs no shell access: ```go s := []metrics.Sample{{Name: "/sched/threads:threads"}} metrics.Read(s) threads := s[0].Value.Uint64() ``` **2. `GODEBUG=schedtrace=1000`.** Set this in the environment and the runtime prints one scheduler summary line per second to standard error, containing `gomaxprocs=`, `threads=`, `idlethreads=` and the run queue lengths. It costs nothing to enable on a canary and it shows the trend, not just a snapshot. Adding `scheddetail=1` prints per-M and per-P detail, which is very verbose. **3. The operating system.** On Linux, `Threads:` in `/proc/<pid>/status` is authoritative, as is counting entries in `/proc/<pid>/task`, or `ps -o nlwp -p <pid>`. This view includes threads the Go runtime did not create — for example threads started internally by a C library you linked against — so it can legitimately exceed the runtime's own count. **4. `/debug/pprof/threadcreate`.** Import `net/http/pprof` and this endpoint reports stack traces at thread creation. Its total is a count of *creation events*, not of live threads, so treat it as attribution ("who made them") rather than as a census ("how many are there"). ## Reading the two numbers together A thread count on its own is close to meaningless: a healthy Go service commonly runs a handful more threads than `GOMAXPROCS` because some goroutines are sitting in system calls. What carries signal is the **relationship over time**: - Both numbers flat, threads in the low tens — normal. - Goroutines climbing, threads flat — that is a goroutine problem (work piling up or goroutines never finishing), not a thread problem. - Threads climbing while goroutines stay flat — the interesting one. Work is not increasing, yet the runtime keeps needing fresh Ms, which means existing Ms are stuck somewhere the runtime cannot take them back, typically a long blocking system call or a call into C code. - Threads that go up and never come down — expected. The Go runtime does not aggressively reap idle Ms, so a burst leaves a permanently higher floor. That last point is why the metric is worth graphing at all: thread count behaves like a high-water mark, and the peak is what you pay for in memory. ## Why it costs anything Each OS thread carries kernel bookkeeping plus a system stack. In a pure-Go program the runtime allocates that stack itself and it is small. When C code is involved, threads are created through the platform's own thread API and get the platform's default stack reservation — commonly 8 MB of virtual address space per thread on Linux — so a thousand threads is gigabytes of address space and a resident-memory cost that grows as those stacks are actually touched. A thread count that has quietly reached four figures is therefore a memory story as much as a scheduling one. ## What to do with the numbers Export both, on the same dashboard, next to resident memory. Alert on the ratio and the trend rather than an absolute threshold, because the healthy absolute value depends on how much blocking work the service does. When the thread line moves and the goroutine line does not, go looking for blocking calls, not for a goroutine leak.