Why can't you average the p99 latencies of ten replicas to get the fleet's p99?

## Why the arithmetic has no meaning A percentile is an **order statistic**: sort every observation and read the value at a given position. Sums and means are defined on the values themselves; a percentile is defined on the *ranking* of a particular set. Combine two sets and the ranking changes in a way that no arithmetic on the two old positions can reconstruct, because the numbers that decide the new position — how many observations each set contributed, and where they sat relative to each other — were thrown away when each set was reduced to a single quantile. The weighting problem alone is enough to kill it. Take three replicas of a district-heating billing service over one window: | Replica | Requests | Reported p99 | | --- | --- | --- | | A | 41,208 | 180 ms | | B | 39,884 | 172 ms | | C | 1,143 | 41 ms | The average of the three p99s is 131 ms. But C, which has just come back from a restart and served 1.4% of the traffic, casts a third of the vote. Pool the 82,235 requests and the slowest 822 of them come almost entirely from A and B, so the real pooled p99 sits close to 175 ms. The dashboard is roughly a quarter low, and it is *systematically* low in exactly the situation you care about: during a rollout, when fresh replicas with little traffic and easy latency are averaged in beside loaded ones. ## What is actually true about the bracket One useful fact survives. The pooled q-quantile always lies between the smallest and the largest of the per-group q-quantiles. The argument is one line: if `x` is the largest of the replica p99s, then every replica has at least 99% of its requests at or below `x`, so their union does too, so the pooled p99 cannot exceed `x`. The mirror argument gives the lower bound. So: - **max of the per-replica p99s** — a genuine upper bound on the pooled p99, and often the number you actually want, because it answers "is any replica serving a bad tail?" - **min of the per-replica p99s** — a genuine lower bound, rarely useful. - **mean of the per-replica p99s** — a number inside the bracket with no interpretation, and one whose error grows exactly as traffic becomes uneven. ## The correct aggregation Do the combining one level down, on the counts, because counts over disjoint sets of observations are additive. 1. Take the per-boundary bucket counts from each replica for the same window and the same bucket grid. 2. Add them boundary by boundary to get the pooled distribution. 3. Estimate the quantile once, from the merged counts. This is why an aggregatable histogram is worth its cost: the grouping decision is deferred to query time, so the same stored data answers the p99 per replica, per region and for the whole fleet without anyone precomputing those combinations. It also imposes a requirement people discover the hard way — the replicas must share a bucket grid. Merge counts from two different grids and the addition is meaningless. ## The same mistake in the time dimension Averaging a series of per-minute p99 values into an hourly figure is the identical error with the groups being minutes rather than replicas. A quiet minute at 03:00 weighs as much as the busy minute in which the tail actually blew out, so the smoothing hides the event. The fix is the same: merge the bucket counts across the hour and estimate once. Be careful that these are two different questions, both legitimate: - *The p99 of the hour's requests* — merge the counts over the hour, estimate once. This is what a user-facing objective usually means. - *The worst minute's p99* — compute per minute, then take the maximum. This is what an alert usually means. A dashboard that averages per-minute p99s answers neither, and it always looks better than both. ## And across services The same instinct produces end-to-end numbers by adding one service's p99 to the next service's p99. That is only right if the same request was in the tail at every hop, which is generally false — most slow requests are slow at one hop. It is not a dependable bound in either direction, so do not treat the sum as conservative. If you need an end-to-end tail, measure end to end: instrument a histogram at the entry point over the whole request, and use per-hop histograms to explain where the time went rather than to reconstruct the total. The general rule worth stating out loud in an interview: **aggregate the counts, never the quantiles.** Any time a percentile appears on the left of an arithmetic operator, something has gone wrong.