skip to content

What is the Go runtime's sysmon thread, and why does it run without holding a P?

level: middleimportance: nice to knowfreq 24%

answer

  1. it is a thread, not a goroutine
  2. it never appears in a goroutine dump
  3. who watches the scheduler when it is saturated
  4. a P is a permit it deliberately does not hold
  5. sleeps microseconds to about ten milliseconds

basics

~20 s

sysmon is a monitoring thread the Go runtime starts at program start and runs for the process lifetime. It holds no P, so GOMAXPROCS does not bound it and it can still act when every P is busy: polling the network, retaking Ps and forcing a periodic collection.

solid answer

~50 s

sysmon is a dedicated OS thread the runtime starts during initialization. It loops forever, sleeping between rounds, and does the housekeeping that nothing else can be relied on to do: polling the network poller if nobody has polled it recently, retaking a P from a goroutine that has been in a system call or running too long, and forcing a garbage collection if none has happened for a couple of minutes. It runs on an M with no P attached, which is the whole point: a P is the permit to run Go code, and if sysmon needed one it would be queued behind exactly the goroutines it exists to supervise. Being outside the scheduler lets it observe a P that is stuck. It also means your process has at least one more OS thread than GOMAXPROCS suggests, even for a trivial program.

go deeper

for a junior

Know that the runtime keeps a background monitoring thread of its own, which is one reason a Go process shows more OS threads than you started goroutines or set GOMAXPROCS to.

for a middle

Explain what a P is, that sysmon holds none, and why that independence matters: it must act precisely when every P is occupied. Name a couple of its duties, such as network polling and forcing a periodic collection.

for a senior

Use it when reading thread counts and stalls in production: sysmon explains part of the gap between GOMAXPROCS and threads, and it is the reason I/O wakeups and periodic collection still happen under a fully saturated scheduler.

for a principal

Treat the runtime's own threads as fixed overhead in capacity planning, and resist tuning proposals that assume GOMAXPROCS bounds threads or that the monitor can be configured away.

## What sysmon is The Go runtime starts one special thread during startup, conventionally called sysmon (system monitor). It is not a goroutine and it does not appear in a goroutine dump. It is an OS thread running a loop inside the runtime for as long as the process lives. ## Why it holds no P In Go's scheduler, a P is the permit to execute Go code; `GOMAXPROCS` sets how many exist, so at most that many goroutines run Go code at once. sysmon runs on a thread with **no P attached**, and that is deliberate. Consider what would happen otherwise. sysmon's job is to notice that a P has been sitting in a system call, that a goroutine has been running for a long time, that nobody has polled the network for a while, or that the heap has not been collected for minutes. Every one of those conditions is a situation in which the scheduler is saturated or stuck. A monitor that needed a scheduling permit would be queued behind precisely the work it is supposed to rescue: the deadlock-adjacent case where every P is occupied is the case where it must run. Running without a P makes sysmon independent of the scheduler it supervises. The cost is that it must work only with runtime internals that are safe to touch without a P. ## What it does each round - **Network polling.** If nothing has polled the network poller recently, sysmon does, so goroutines waiting on I/O become runnable even when every P is busy with CPU work. - **Retaking Ps.** A P whose goroutine has been in a system call for a while, or has been running without yielding, can be taken back so other goroutines get a turn. The detailed preemption machinery is a topic of its own; the point here is that the observation comes from outside the scheduler. - **Forcing collection.** If no garbage collection has run for about two minutes, sysmon triggers one, so a mostly idle program still returns memory and still runs finalisers and cleanups. ## Its duty cycle sysmon does not spin. It sleeps between rounds, starting at tens of microseconds and backing off towards about ten milliseconds when the program is idle, so an idle Go process is not burning a core on monitoring. When the program is busy it polls at the short end of that range. ## The consequence people actually meet Count the threads of a trivial Go program and there are several, more than `GOMAXPROCS` would suggest. `GOMAXPROCS` bounds Ps, not threads: the number of goroutines executing Go code simultaneously, not the number of OS threads the process holds. sysmon's thread is one of the extras and it is always there. Treating `GOMAXPROCS` as a thread limit is a common misreading and leads people to chase a thread count that was never going to match it. ## What sysmon is not It is not the garbage collector. The collector's mark work runs on ordinary worker goroutines that do consume Ps; sysmon merely notices when too long has passed and asks for a cycle. It is also not a goroutine you can see or profile, so goroutine dumps and goroutine counts do not include it. And it is not user-configurable: there is no knob to disable it, and the right mental model is that it is part of the runtime's floor cost, like the scheduler itself. ## Why an interviewer asks The question separates people who have read `GOMAXPROCS` as a CPU knob from people who understand the G/M/P split. If you can say that a P is a permit to run Go code, that sysmon deliberately holds none so it can act when all permits are taken, and that this is one reason thread count exceeds `GOMAXPROCS`, you have demonstrated the model rather than the vocabulary.

  • Why can't the runtime do sysmon's work in an ordinary goroutine?
    A goroutine needs a P to run. If every P is held by a goroutine that is not yielding, or is tied up in a system call, the monitoring goroutine would not be scheduled at exactly the moment it is needed. Holding no P keeps sysmon independent of the scheduler state it is supposed to observe.
  • Does sysmon count against GOMAXPROCS?
    No. GOMAXPROCS bounds the number of Ps, which caps how many goroutines execute Go code simultaneously. It never capped OS threads. sysmon's thread has no P, so it sits outside that budget entirely, which is one reason a process always shows more threads than GOMAXPROCS.
  • Does sysmon spin and burn a core on an idle program?
    No. It sleeps between rounds and backs off as the program goes quiet, from tens of microseconds up to roughly ten milliseconds, tightening again when there is work to watch. An idle Go process therefore costs a negligible amount of CPU for monitoring.

It is the night watchman with his own keys rather than a member of the shift. If he had to take a slot on the production line, he would be stuck waiting exactly when the line jammed.

saying these in an interview costs you the question

  • Calls sysmon an ordinary goroutine scheduled like any other
  • Says GOMAXPROCS caps the number of OS threads
  • Believes sysmon spins constantly and burns a core
  • Confuses sysmon with the collector's background mark workers
  • Expects to see sysmon in a goroutine dump