Which aggregations across a fleet of per-process gauges are meaningful, and why is an average of averages wrong?

## A gauge is a point sample of a level A gauge records where something stood at the instant it was observed. It accumulates nothing, so unlike a counter it carries no memory of what happened between one observation and the next, and that shapes every aggregation you can build on top of it. When the same gauge is published by every process in a fleet, you hold a grid: one value per process per timestamp. Collapsing that grid into a single fleet number is where most dashboards quietly go wrong, because the collapse that is arithmetically available is not always the collapse that means something. ## Which fleet aggregations mean something The test is not about the metric system; it is about the physical quantity. Ask whether the thing being measured **adds up**. - An **extensive** quantity is proportional to the size of what you measured: bytes of heap in use, open connections, items waiting in a queue, threads alive. Two processes holding 400 MB each are genuinely holding 800 MB. Summing across the fleet is meaningful, and so is the mean, the maximum and the minimum. - An **intensive** quantity is a ratio or an intensive property that does not scale with size: utilisation percentage, a cache hit ratio, a mean object size, a temperature. Summing fourteen hosts each at 46% utilisation produces 644%, a number with no physical referent at all. | Aggregation | Extensive gauge (queued items) | Intensive gauge (hit ratio) | |---|---|---| | Sum across processes | Meaningful: total work waiting | Meaningless: no physical referent | | Mean across processes | Meaningful: typical per-process level | Misleading unless every process carries equal weight | | Max / min across processes | Finds the outlier process | Finds the outlier process | | Count of processes over a threshold | Tells you how widespread it is | Tells you how widespread it is | Two further cautions apply to both columns. First, a fleet sum moves when the fleet size moves, so a rolling deploy that briefly runs eleven processes instead of fourteen produces a dip that is an artefact of the deploy, not of the workload; wherever that matters, publish the process count beside the sum so the two can be read together. Second, a max over a window is the maximum of the samples you happened to take, not the true maximum of the underlying level — a spike between two observations leaves no trace at all. ## Why an average of averages is not the average The most common intensive-quantity mistake has its own name. When each process publishes a value that is *already* a ratio or an average, taking the unweighted mean of those values across the fleet gives every process an equal vote regardless of how much work it did. That is only the fleet figure in the special case where the workload is perfectly even, which is precisely the case that stops holding during an incident. The correct fleet ratio recombines the parts: 1. Have each process publish the **numerator** and the **denominator** separately, as counters — cache hits and cache lookups, not the ratio. 2. Sum each of those across the fleet for the window you care about. 3. Divide once, at query time, at the level you want to report. That way the arithmetic weights each process by its actual volume, and the same two series still support a per-process breakdown when you want one. ## A worked example A seed-catalogue ordering service ran fourteen processes behind a load balancer and each published a `cache_hit_ratio` gauge. During a 5,400-request-per-second peak, a sticky-session bug pinned one process to 3,860 requests per second while the other thirteen shared the remaining 1,540. The busy process, thrashing its cache, sat at a 0.61 hit ratio; the thirteen quiet ones sat at 0.96 each. The fleet dashboard averaged the fourteen gauges and showed **0.94** — a healthy-looking green number that stayed green for the whole incident and that nobody could reconcile with what users were experiencing. The true fleet ratio weights by volume: `(3860 x 0.61 + 1540 x 0.96) / 5400`, which is about **0.71**. More than a quarter of all lookups were missing, and the dashboard was off by twenty-three percentage points. Nothing was broken in the metrics pipeline; the ratio had been divided too early, inside each process, and once divided it could not be recombined. ## What to publish instead - **Publish the raw parts, divide at query time.** Any value you compute inside a process and then publish as a single number can no longer be re-aggregated. Ratios, percentages and means all have this property. - **Keep an identifying dimension on the series** so the per-process values are still reachable when you want to find the outlier — the fleet number and the outlier hunt need different aggregations. - **Say the aggregation out loud on the dashboard.** "Mean of per-process hit ratio" and "fleet hit ratio" are different numbers, and a panel titled just "hit ratio" invites the reader to assume the one that would have been correct. - **Prefer max over mean for saturation.** One process at 98% utilisation is an incident even when the fleet mean is 40%, and the mean is designed to hide exactly that.