skip to content

How does a Go P look for work when its own local run queue is empty?

level: middleimportance: must knowfreq 48%

answer

  1. it does not park straight away
  2. the shared queue and I/O come first
  3. then it goes shopping among the others
  4. half a victim's queue, several passes
  5. random victims, spinning before sleeping

basics

~20 s

It drains a batch from the global run queue, polls the network poller, then steals: up to four randomised passes over the other Ps, taking about half of a victim's local run queue. Only after all of that does its thread park.

solid answer

~50 s

When a P's `runnext` and local run queue are both empty, its thread enters the runtime's find-work loop. It first drains a batch from the global run queue — roughly that queue's length divided by the number of Ps — then does a non-blocking poll of the network poller for goroutines whose I/O has completed, and only then starts stealing. It marks its thread as spinning and makes up to four passes over the other Ps in a randomised order, grabbing about half of a victim's local run queue in one operation, running one of those goroutines directly and queueing the rest. On the final pass only it will also take a victim's timers and its `runnext`, pausing briefly first. If everything is empty it releases the P to the idle list and re-checks all the sources once more before parking, which is what stops work submitted during the decision to sleep from being missed.

go deeper

for a junior

Know that an idle P does not simply wait for work to be handed to it: the runtime takes runnable goroutines from busier Ps to fill it.

for a middle

Be able to give the order out loud — local, global, network poller, then stealing from randomly chosen Ps, and only then parking the thread — and say how much a steal takes.

for a senior

Expect to reason about the latency this costs: work sits on the P that created it until a thief finds it, which shows up as uneven CPU use under bursty, short-task load.

for a principal

Own the framing that scheduler imbalance is judged end to end, not per worker, and that restructuring an application to 'spread work' rarely beats letting the runtime steal it.

## The find-work loop A P whose `runnext` slot and 256-entry local run queue are both empty does not park its thread. It enters the runtime's find-work loop, which tries a fixed sequence of sources and only sleeps when all of them come up empty. The order is what an interviewer is listening for. **1. The global run queue.** The P takes a *batch*, not one goroutine: roughly the queue's length divided by the number of Ps, plus one, capped at half its own local queue's capacity. One goroutine is run immediately and the rest go into the local queue. Taking a batch means the scheduler's global lock is acquired once for many goroutines. **2. The network poller.** A non-blocking poll of the runtime's integrated poller returns goroutines whose file or socket I/O has completed. Doing this before stealing matters: I/O-completed goroutines are genuinely ready work that nobody else is holding. **3. Stealing from other Ps.** Now the thread marks itself *spinning* and makes up to four passes over all the other Ps. The visiting order is randomised — a random starting point and a stride chosen to be coprime with the number of Ps, so every P is visited exactly once per pass without a fixed pattern that would make several thieves collide on the same victim. On each victim it grabs about **half** of the victim's local run queue in one operation, moving the batch into its own queue and running one of the stolen goroutines directly. On the **final** pass only, it will also look at the victim's timers and at the victim's `runnext` slot, pausing briefly first so the owner has a chance to run that goroutine itself. Taking the fast slot first would defeat the purpose the fast slot exists for. **4. Re-check, then park.** If nothing was found, the thread gives its P up to the idle list and then checks everything again — the global queue, every P's local queue, the poller and pending timers — before it actually parks. ## Spinning threads and the race the re-check closes A *spinning* M is a thread that is actively hunting rather than sleeping. The state exists to close a window: a thread that looked everywhere, found nothing and went to sleep would be useless if a goroutine were readied a nanosecond later, because the readying side only wakes a thread when it believes none is searching. Keeping a searcher alive, and forcing it to re-check after declaring itself no longer spinning, guarantees that newly submitted work is either seen by an existing searcher or triggers a wake. Spinning is capped, though, because searching burns CPU. The runtime starts a new spinning thread only when none is currently spinning, and it declines to spin when the number of spinning threads is already about half the number of busy Ps. On a machine with many cores running a program with little parallelism, uncapped searching would show up as real CPU usage doing nothing. ## Why this is not the same as a general work-stealing scheduler The idea of stealing is not Go's invention, but this arrangement is Go's: the specific ordering of local, then global, then poller, then victims; the batch drained from the global queue; the four randomised passes; taking half a queue; and treating `runnext` and timers as last-pass-only prizes. Those constants and that ordering are what the question is about. ## What it means in production - **An idle CPU next to a backlog is expected, briefly.** Work reaches another P only after a thief finds it, which costs a search. For microsecond-scale tasks that latency is a visible fraction of the work; for millisecond tasks it disappears into the noise. - **You cannot pin a goroutine to a P.** `runtime.LockOSThread` pins a goroutine to a *thread*, which is a different guarantee and does not stop the queue mechanics described here. - **Uneven per-worker throughput in a pool is not evidence of a bug.** Work has affinity to the P that created it, and stealing pulls the imbalance back gradually rather than instantly. - **A thread that is spinning is not stuck.** Seeing threads burning small amounts of CPU in the scheduler's search path under a light load is normal behaviour, not a leak.

  • What is a spinning thread in Go's scheduler, and why does the runtime limit how many there are?
    A spinning M is a thread actively hunting for work — checking queues, the poller and other Ps — instead of sleeping. Keeping one alive means newly readied work is picked up without a wake-up, and it must re-check everything before parking. But searching burns CPU, so the runtime starts a new spinning thread only when none is spinning and declines to spin once about half the busy Ps already are.
  • Where do the goroutines a thief takes actually end up?
    The steal moves the batch into the thief's own local run queue and returns one goroutine to run immediately, so the thread does not go round the find-work loop again. Nothing is placed in the thief's `runnext`. The victim keeps the other half of its queue, and keeps its `runnext` unless this was the thief's last pass.
  • If every P is empty, what does the thread do before it sleeps?
    It gives up its P to the idle list, then re-checks all the sources — the global run queue, each P's local queue, the network poller and pending timers — before actually parking. A thread that was spinning must do this, because a goroutine readied by another thread just after it looked could otherwise sit runnable with nobody scheduled to notice.

saying these in an interview costs you the question

  • Says an empty P immediately parks its thread
  • Claims a background thread rebalances the run queues
  • Thinks a thief takes one goroutine at a time
  • Says Ps steal in a fixed round-robin order
  • Believes the global queue is only read at startup