skip to content

Which Go runtime profiles carry pprof labels, and which ignore them entirely?

level: middleimportance: nice to knowfreq 24%

answer

  1. not every profile knows about tags
  2. time and stacks, not bytes
  3. one census profile also keeps them
  4. CPU and goroutine, and that is all
  5. heap, block and mutex ignore them

basics

~20 s

Only the CPU profile and the goroutine profile carry runtime/pprof labels. The heap and allocation profiles, the block profile, the mutex profile and threadcreate ignore them, so labels cannot split memory or contention by tenant.

solid answer

~50 s

Labels are attached to goroutines, but only two profile types record them: the CPU profile, where each sample carries the tags of the goroutine that was running, and the goroutine profile, where each goroutine's entry carries its current label set. That combination is powerful — it tells you both which tenant is burning CPU and which tenant's work is currently parked. Everything else ignores labels: the heap and allocation profiles, the block profile, the mutex profile and threadcreate all record stacks with no tags, so `-tagfocus` on them matches nothing. If you need per-tenant memory or contention numbers you need a different instrument — your own counters, or a separate process or profiling window per class of work. A related nicety in recent Go: goroutine tracebacks can print a goroutine's pprof labels, which makes a panic dump say whose work was in flight.

go deeper

for a junior

Remember the pair that supports labels: the CPU profile and the goroutine profile. If you filter a heap profile by tag and get nothing back, that profile simply does not record tags.

for a middle

Explain why the two labelled profiles answer different questions — one is time-weighted sampling, the other a snapshot of goroutine counts — and name the profiles that ignore labels entirely.

for a senior

Plan instrumentation around the limit: decide which attribution questions labels can answer and add your own counters or comparison windows for memory and contention, rather than discovering the gap during an incident.

for a principal

Own the choice of what a service measures per class of work, given that only CPU time and goroutine counts come free from labels, and weigh the cost of custom accounting against the value of the numbers it produces.

## The short answer `runtime/pprof` labels are read by exactly two profiles: - **The CPU profile.** Each sample the profiler takes carries the label set of the goroutine that was executing. This is the case the feature was designed for: split one hot function's time by tenant, job kind, endpoint or priority. - **The goroutine profile.** Each goroutine's record carries the labels that goroutine currently holds. This gives you a census rather than a time breakdown: not who is burning CPU, but whose work is currently in flight or parked, and where. Every other profile the runtime produces ignores labels. The heap profile and the allocation profile record allocation stacks and sizes with no tags. The block and mutex profiles record where goroutines waited, again with no tags. So does threadcreate. Applying a tag filter to any of them matches nothing, which is a confusing failure the first time you meet it — the tool does not tell you the profile *cannot* carry tags, it simply returns an empty result as though no work matched. ## Why the two behave differently from each other The distinction matters when you choose which profile to reach for. A CPU profile is time-weighted and sampled: a tenant that appears in 30% of samples was using roughly 30% of the CPU during the window. A goroutine profile is a snapshot of counts: a tenant with 4,000 goroutines has 4,000 goroutines, whether they are running hot or blocked on a network read. For attribution work you often want both. If one tenant dominates the CPU profile, that tenant is expensive. If one tenant dominates the goroutine profile but barely appears in the CPU profile, that tenant's work is mostly waiting — a very different conclusion, and one you can only draw because both profiles preserve the tags. ## What to do about the profiles that ignore labels The most commonly wanted missing one is memory. There is no supported way to make the heap profile attribute bytes to a tenant via labels. Realistic alternatives: - **Instrument what you care about directly.** Counters or histograms of bytes processed per class of work are usually what the question actually needs, and they are exact rather than sampled. - **Separate the windows.** Profile while a single class of work is running, or in an environment where only one tenant's jobs are scheduled, and compare heap profiles between windows. Crude, but it isolates the variable. - **Compare profiles rather than tag them.** Taking a heap profile before and after a change, or between two windows with different traffic mixes, extracts a difference without needing tags at all. For contention, the block and mutex profiles have the same limitation, and the same workaround applies: change the mix and compare, or count in your own code. ## Labels in tracebacks One further place labels can surface: **goroutine tracebacks**. As of **Go 1.27**, a traceback prints a goroutine's `runtime/pprof` labels for modules that declare `go 1.27` or later, and `GODEBUG=tracebacklabels=0` turns that off. This is a small change with a real operational payoff — a panic dump or a stack dump then says whose work the goroutine was doing, not merely which functions it was in, which is exactly the attribution question you are asking at the moment something crashes. ## Practical consequences Three things follow for how you instrument a multi-tenant Go process: 1. Do not build a plan around per-tenant heap or mutex numbers coming from labels. Decide up front which questions labels can answer (time and goroutine counts) and which need other instrumentation. 2. When a tag filter returns an empty profile, check the profile type before you go looking for bugs in your labelling. An empty result from a heap profile is expected; an empty result from a CPU profile means labels really were not applied where the work runs. 3. Keep the label vocabulary consistent between the CPU and goroutine profiles, because the pairing of the two is where most of the diagnostic value is: the same tenant key must mean the same thing in both, or you cannot line them up.

  • You want per-tenant heap usage in a Go service. Why will labels not give it to you, and what would?
    The heap and allocation profiles record allocation stacks and sizes with no tags, so a tag filter over them matches nothing. Practical alternatives are instrumenting bytes per class of work with your own counters, profiling windows in which only one class of work runs, or comparing heap profiles across two windows with different traffic mixes to extract the difference.
  • What does a tenant dominating the goroutine profile but barely appearing in the CPU profile tell you?
    That the tenant's work is mostly waiting rather than computing — many goroutines parked on I/O, locks or channel operations while consuming little CPU. The goroutine profile is a count snapshot and the CPU profile is time-weighted, so having labels on both lets you separate "expensive" from "numerous", which is usually the first fork in a capacity conversation.
  • What did Go 1.27 change about labels and tracebacks?
    Tracebacks now include a goroutine's runtime/pprof labels for modules that declare `go 1.27` or later, so a panic dump or stack dump says whose work the goroutine was doing rather than only which functions it was in. `GODEBUG=tracebacklabels=0` turns the behaviour off if the extra output is unwanted.

saying these in an interview costs you the question

  • Expects labels to split a heap profile by tenant
  • Believes every runtime profile carries the goroutine's tags
  • Reads an empty tag-filtered heap profile as a labelling bug
  • Confuses the goroutine profile's counts with CPU time
  • Assumes the mutex profile can attribute contention per tenant