Goroutine Labels and Attribution

## The problem labels solve A CPU profile answers *where* a Go program spent time: which functions, reached up which call stacks. In a process that does the same work for many callers — a background job runner executing jobs on behalf of many customers inside one binary — that is only half the question. The profile may say, correctly and uselessly, that 40% of CPU is inside one decode function. Every tenant's jobs go through that function. The profile cannot say which tenant's work put it there, because a stack trace has no notion of *on whose behalf*. `runtime/pprof` labels add that dimension. A label is a string key with a string value. Labels are attached to a **goroutine**, and the runtime copies whatever labels a goroutine currently carries onto each profile sample it takes from that goroutine. Filtering and grouping by those tags then happens in the analysis tool. ## The API Three calls matter. **`pprof.Labels(args ...string) LabelSet`** builds a label set from alternating keys and values: ```go labels := pprof.Labels("tenant", "acme", "kind", "reindex") ``` It panics if you pass an odd number of arguments — there is no compile-time check that pairs are balanced, so a missing value is a runtime panic, not a silent empty label. **`pprof.Do(ctx context.Context, labels LabelSet, f func(context.Context))`** is the call you should normally use. It derives a context from `ctx` with `labels` added, sets those labels on the goroutine that called it, invokes `f` with the derived context, and — this is the part that makes it safe — restores the goroutine's previous label set when `f` returns, including when `f` panics its way out. Because it is scoped like a `defer`, you cannot forget to unset labels. ```go func handle(ctx context.Context, j Job) { pprof.Do(ctx, pprof.Labels("tenant", j.Tenant, "kind", j.Kind), func(ctx context.Context) { execute(ctx, j) }) } ``` **`pprof.SetGoroutineLabels(ctx context.Context)`** is the lower-level form: it sets the current goroutine's labels to whatever the context carries, with no scope and no restore. It exists for the cases where a callback shape does not fit — typically a long-lived loop that re-labels itself each iteration. Combine it with `pprof.WithLabels(ctx, labels)`, which returns a derived context carrying the labels without touching any goroutine. You can read labels back out of a context with `pprof.Label(ctx, "tenant")`, which returns the value and an `ok` boolean, and iterate them with `pprof.ForLabels(ctx, func(key, value string) bool)`. Both read the **context**, not the goroutine, which is a useful reminder of how the two halves relate: the context is how labels travel through your code, and `Do`/`SetGoroutineLabels` are what actually apply them to the runtime. ## What you get downstream Once samples carry tags, `go tool pprof` can work with them: the interactive `tags` command lists the label keys and values present in the profile with their sample weight, and flags such as `-tagfocus` and `-tagshow` narrow the profile to particular tag values or restrict which keys are displayed. That is how the useless "40% in decode" becomes "of that 40%, three quarters is one tenant's reindex jobs". ## Cost and cardinality Labels are not free but they are cheap: a label set is an immutable structure stored on the goroutine, and `Do` costs an allocation plus the save/restore around the call. The real cost is cardinality. A label value per *class* of work — tenant, job kind, priority, endpoint — groups thousands of samples into a handful of buckets and is what the tooling is designed for. A label value per individual unit of work, such as a job UUID or a request id, gives you a profile where almost every sample has a unique tag: it inflates the profile, slows the tool, and aggregates into nothing. Labels are a grouping dimension, not a tracing identifier; if you need per-request identity, that is a different instrument. ## Common first mistakes Applying `Do` at process start rather than around each unit of work labels nothing useful. Assuming a label set will appear in a heap profile does not work — not every profile type carries labels. And treating labels as something the context alone propagates is wrong: putting labels in a context and passing it somewhere does not label the goroutine on the other end until that goroutine applies them.