How do you use `go tool pprof -diff_base` on two saved CPU profiles to find what got slower between builds?

## The mechanics pprof can subtract one profile from another. Given two files it loads the positional one as the profile under study and the flag's argument as the base: ``` go tool pprof -diff_base=build-4821.pprof build-4899.pprof ``` Every sample stack present in both contributes its difference; stacks only in the new profile contribute their full weight positive; stacks only in the base contribute negative. From then on every view — `top`, `list`, `peek`, the flame graph — displays deltas rather than absolutes, and rows can be negative. There are two subtraction flags and they differ in what the percentages are relative to. `-base` subtracts and computes percentages against the resulting difference. `-diff_base` subtracts and computes percentages against the *base* profile, so a row reading 14% means it grew by 14% of the old total — usually the more interpretable framing when you are asking how much worse a build got. ## Why this is a senior question rather than a flag question The subtraction is trivial; the interpretation is where people go wrong. **Comparability.** A profile is a sample of one run of one workload. Subtracting two profiles is only meaningful if the runs were alike: the same inputs (for a build-step code generator, the same schema files, not last week's tree), a comparable duration, the same machine class and the same number of CPUs. A profile collected over 30 seconds against one over 10 will show every row growing, and none of that is a regression. If durations differ, scale the base to match before reading it — pprof exposes a normalisation option for exactly this — but the safer discipline is capturing both under the same harness so no scaling is needed. **Noise.** Samples are a statistical estimate. On a short capture, a row moving by a couple of percent is inside the noise, and chasing it wastes the day. The cheap defence is a third capture: run the base build twice and diff it against itself. Whatever shows up in *that* diff is your noise floor, and only deltas comfortably above it are real. **Attribution shifting.** Between builds the compiler's inlining decisions can change, and a function that used to be inlined into its caller suddenly appears in its own right — a big positive row next to a big negative row with no change in total cost. Reading the diff at cumulative rather than flat weight, or grouping at package granularity first, makes these cancel out and keeps you looking at genuine movement. **What a CPU diff cannot see.** It only accounts for time on a CPU. If the new build got slower because it waits longer — on a lock, on a channel, on I/O — the CPU profiles can be almost identical while wall-clock latency doubles. A flat CPU diff in the face of a real slowdown is itself a finding: it tells you the cost moved off-CPU and that a CPU profile is the wrong instrument for the next step. ## A workable procedure 1. **Keep the artifacts identifiable.** A directory of profiles, one per build, is only useful if each carries the commit, the toolchain version, the input corpus and the capture duration. A profile with no provenance cannot be diffed against anything with confidence, and this is the single most common reason the file tree of saved profiles goes unused. 2. **Establish the noise floor** by diffing two captures of the same build. 3. **Diff the suspect pair** with `-diff_base` and read `top -cum` first: which subtree grew. 4. **Narrow** with `peek` to see whether the growth arrives from a new caller, and with `list` to see whether a specific line grew. 5. **Confirm** by recapturing both, or by bisecting builds if the delta straddles many commits. 6. **Verify the fix the same way**, diffing after against before — which is the same command with the arguments in the other order. ## The honest limits A diff localises movement; it does not explain it. A row that grew may have grown because it is genuinely slower, because it is called more often, because work moved into it from somewhere else, or because the input got bigger. Only the first of those is a code regression, and distinguishing them is the work `peek` and the surrounding context do. Treat the diff as a pointer, and always confirm the story against the code change between the two builds before you act on it.