skip to content

Which goroutines inherit runtime/pprof labels, and when do those labels go away?

level: middleimportance: should knowfreq 42%

answer

  1. set at spawn, not at send
  2. a child copies the parent's set
  3. a snapshot, not a live link
  4. Do restores only its own goroutine
  5. a goroutine already running gets nothing

basics

~20 s

A goroutine inherits the labels its creator carried when the go statement ran, and keeps them for life unless it sets its own. Already-running goroutines get nothing, and pprof.Do restores labels only on its own goroutine.

solid answer

~50 s

Labels live on the goroutine, not on the context. When a `go` statement runs, the new goroutine copies whatever label set its creator carries at that instant, and it keeps that set for its whole life — even after the parent's `pprof.Do` has returned and restored the parent's own labels. That has two consequences. Goroutines started *inside* a labelled region are attributed correctly for free, which is what makes labelling at the spawn point work. But a goroutine that was already running when you labelled — a long-lived worker started at process boot that later picks work off a channel — inherits nothing, and no context you send it will change that. For those you must re-label inside the loop, either with `pprof.Do` around the body or with `pprof.SetGoroutineLabels(pprof.WithLabels(ctx, labels))` at the top of each iteration. Passing a labelled context alone never labels the receiving goroutine.

code

go · 7 lines
go
// started at process boot, so it inherits no labels
for j := range incoming {
	pprof.Do(ctx, pprof.Labels("tenant", j.Tenant), func(context.Context) {
		execute(j)
	})
	// labels restored here, so idle time is attributed to nobody
}

go deeper

for a junior

Remember the direction: a goroutine gets labels only from the goroutine that started it, at the moment it was started. Nothing you pass over a channel changes an already-running goroutine's labels.

for a middle

Explain the snapshot rule and the asymmetry of pprof.Do: it restores labels on its own goroutine only, so children keep what they inherited. Show the fix of labelling inside the work loop.

for a senior

Read the symptom from a real profile: untagged samples mean the labels were applied where the work is not, and one tenant dominating background work means a long-lived goroutine inherited that tenant's tags and kept them.

for a principal

Set the house rule for where labels are applied so profiles from different services are comparable, and treat any goroutine that outlives its creating unit of work as required to set its own labels.

## Two places labels can live It is easy to think of `runtime/pprof` labels as living in a `context.Context`, because that is where the API puts them. They live in both places, and the distinction decides every question about inheritance: - **The context** is how labels travel through your own code. `pprof.WithLabels(ctx, labels)` returns a derived context carrying a label set; `pprof.Label(ctx, key)` and `pprof.ForLabels(ctx, f)` read them back. - **The goroutine** is what the profiler reads. When the runtime takes a sample, it copies the label set attached to the *goroutine* onto that sample. It never looks at any context. `pprof.Do` bridges the two: it derives the context and applies the labels to the current goroutine. `pprof.SetGoroutineLabels(ctx)` does only the second half. This is why sending a labelled context to another goroutine over a channel labels nothing — the receiving goroutine's own label set is untouched until it calls one of those two functions. ## The inheritance rule One rule covers creation: **a new goroutine inherits the label set of the goroutine that created it, as of the moment the `go` statement executes.** It is a snapshot, not a link. Consequences: 1. Work spawned inside `pprof.Do` is attributed automatically. If a labelled function fans out into several helper goroutines, every one of them carries the same tenant tag, and so does anything they spawn in turn. Whole subtrees of work get attributed from one call at the top. 2. The inherited set is **not** unwound when the parent's `Do` returns. `Do` restores labels on the goroutine that called it and on no other. A goroutine started inside a labelled region and left running keeps that tenant's labels indefinitely — so a long-lived goroutine spawned during one tenant's request will attribute all its future work, for every tenant, to that first one. If a spawned goroutine outlives the unit of work that created it, either re-label it or accept that its samples are mislabelled. 3. Goroutines that already existed inherit nothing, ever. Labelling later changes nothing about them. ## The attribution failure this causes The common shape in a multi-tenant job runner is a fixed set of long-lived executor goroutines started at process boot, fed units of work as they arrive. Labelling looks like it should go where those goroutines are created — that is where the `go` statements are. It does not work: at boot there is no tenant yet, so every executor carries an empty label set for life, and the profile comes back exactly as undifferentiated as it was before you instrumented anything. Then `-tagfocus` on a tenant matches no samples at all and the instrumentation looks broken. The fix is to label where the *work* is, not where the *goroutine* is: ```go for j := range incoming { pprof.Do(ctx, pprof.Labels("tenant", j.Tenant), func(context.Context) { execute(j) }) } ``` Because `Do` restores the previous (empty) set at the end of each iteration, an executor sitting idle between units of work is correctly attributed to nobody, and the next unit of work gets its own tenant. When the loop body is not conveniently a function — a long `select` with several arms, or a hot loop where you would rather not pay for a closure — `pprof.SetGoroutineLabels(pprof.WithLabels(ctx, labels))` sets the goroutine's labels directly. It does not restore anything, so you own the unset: label at the top of every iteration so that the previous tenant's tags are always replaced rather than lingering. ## Reading the situation from a profile Two symptoms are worth recognising. Samples with no tags at all, from goroutines you believe you labelled, mean the labels were applied to a goroutine that never runs the work — usually the spawner, or the process at boot. Samples where one tenant's tag is implausibly dominant, particularly on background work like flushing or compaction, usually means a long-lived goroutine inherited that tenant's labels at creation and never let go of them. ## Practical guidance Apply labels as close to the unit of work as you can, prefer `pprof.Do` so the scope is enforced for you, and treat any goroutine that outlives its creating unit of work as needing its own labels rather than inherited ones. And do not rely on context propagation alone: a `context.Context` moving between goroutines carries the labels for your code to read, but only `Do` or `SetGoroutineLabels` makes the profiler see them.

  • A goroutine is started inside a pprof.Do call and keeps running afterwards. What are its labels an hour later?
    Still the labels it inherited at the `go` statement. `Do` restores the label set only on the goroutine that called it; a child's copy is independent and is never unwound. So a background goroutine spawned during one tenant's work will attribute everything it ever does to that tenant until it sets its own labels. Long-lived children should re-label themselves rather than inherit.
  • Does sending a context created by pprof.WithLabels to another goroutine label that goroutine?
    No. `WithLabels` only puts the label set in the context; the profiler reads labels off the goroutine, not off any context. The receiving goroutine stays unlabelled until it calls `pprof.Do` or `pprof.SetGoroutineLabels` with that context. Passing the context is how the values travel; applying them is a separate, explicit step.
  • Why does labelling at the point a pool of executor goroutines is created produce an unlabelled profile?
    Because inheritance is a snapshot taken when the `go` statement runs. At process boot there is no tenant yet, so each executor copies an empty label set and keeps it for life; work arriving later never changes it. The labels have to be applied by the goroutine that actually runs the work, inside the loop, once per unit of work.

saying these in an interview costs you the question

  • Thinks labels flow with the context into the receiving goroutine
  • Believes a child goroutine loses its labels when the parent's Do returns
  • Labels the pool at startup and expects per-tenant samples
  • Assumes inheritance is a live link back to the parent's current labels
  • Uses SetGoroutineLabels once and never re-labels the loop