Sampling Runtime Metrics

## Why a snapshot is expensive Go's allocator keeps its counters where allocation happens: on the per-P mcache, so that the fast path needs no shared writes. That design is what makes allocation cheap, and it is also why an accurate global total is not simply sitting in a variable somewhere. Any API that promises *every* number in a struct describes the same instant has to reconcile all of those distributed counters while nothing is changing them. `runtime.ReadMemStats(&m)` makes exactly that promise, and it keeps it the blunt way: it stops the world, walks the runtime's state on the system stack to fill the struct, and starts the world again. Nothing else in your program runs during that window. The pause is typically short — the same order as the runtime's other stop-the-world operations — but it is a real latency event and it is paid by every goroutine, including the ones serving your slowest tail requests. Call it once a second and it disappears into the noise; call it per request, or inside a benchmark loop, and you have built a latency generator whose only output is a number nobody is reading that often. ## What runtime/metrics does instead `runtime/metrics` was introduced to break the tie between *observing* the runtime and *pausing* it. `metrics.Read` does not stop the world. It takes an internal runtime lock, ensures the aggregates that the requested metrics depend on are up to date — reading the runtime's consistent per-P statistics structures, which are designed to be readable without halting the mutators — and then fills each `Sample.Value`. Two properties follow from that design, and both matter when you write a reporting daemon: 1. **Cost scales with what you asked for, not with the whole world.** A metric that is a plain counter is nearly free to read. A metric that requires aggregating heap statistics across all Ps costs more. A histogram costs more again, because its bucket counts have to be produced. 2. **The aggregates are computed once per `Read` call.** Six names in one call share the underlying aggregation and one lock acquisition; six separate calls repeat both. Batching is a genuine optimisation, not style. The lock is the part people forget. `metrics.Read` is cheap relative to a world stop, but it is a process-wide serialisation point, so a goroutine reading it in a tight loop can contend with, say, a debug endpoint doing the same thing. ## Choosing between them For anything you sample on a schedule, `runtime/metrics` is the default. It covers the ground `MemStats` covers — allocation totals, heap occupancy by memory class, GC cycle counts, GC pause distributions — with names that are discoverable at runtime and a documented unit per name. `ReadMemStats` still has a place. If you have existing code or a dashboard built on `MemStats` field names, or you specifically want the struct's own shape (its fixed `PauseNs` ring of recent pause durations, its `BySize` table), you read it — just rarely, on a ticker measured in seconds, never on a request path. A nuance worth stating in an interview: the cost of `ReadMemStats` is not proportional to how much memory you are using in any simple way, and it is not a GC. It does not force a collection, it does not free anything, and it does not allocate. The cost is the stop-the-world round trip plus the work to reconcile per-P state, so it is closer to constant-with-P-count than to anything about heap size. ## The failure mode this prevents The classic incident is self-inflicted: a well-meaning instrumentation layer calls `ReadMemStats` inside a middleware to record heap usage per request. Under load, the process now stops the world thousands of times a second. Latency degrades in a way that looks like GC pressure — because it *is* pause time — and the profile of the offending call is small, because the cost lands on every other goroutine rather than on the sampler. The tell is that pause events line up with request volume rather than with allocation volume, and the fix is to move the measurement to a single reporting goroutine on a ticker and to use `runtime/metrics` for it. ## Summary One API buys consistency across a whole struct by pausing the program; the other buys cheapness by letting you name exactly the numbers you need and reading them without a pause. Sample regularly with the second, reach for the first deliberately and infrequently, and batch either one into a single call from a single goroutine.