Why can a performance run's per-interval 95th-percentile times not be averaged into one figure for the whole run?

## What a percentile is, mechanically A percentile is a **rank statistic**. The 95th percentile of a set of response times is the value you reach after sorting every measured time and walking 95% of the way along the sorted list. It is defined only with respect to *one particular set of samples*. Add samples, remove samples or reweight them and the answer changes, and the only way to learn the new answer is to go back to the samples and rank them again. That is why the arithmetic that works on other summary figures does not work here. A total is a sum over samples, so totals from two windows add. An arithmetic average is a sum divided by a count, so two of them combine correctly as long as you carry both counts. A percentile is neither a sum nor a ratio of sums — it is a **position in a sorted list**, and positions in two separate lists do not combine into a position in the concatenated list. ## Why averaging the per-interval figures is not merely approximate Two independent things go wrong, and repairing one leaves the other standing. 1. **Every interval gets equal weight regardless of size.** A minute that served 12,000 requests and a minute that served 400 each contribute one number to the average. The pooled rank weights every *request* equally instead, so the two answers diverge whenever the request rate varied — which it does during ramp, during a pause, and during exactly the minutes when the system was struggling. 2. **The shape inside each interval has already been discarded.** A minute in which 30% of requests were slow and a minute in which 6% were slow can report almost the same per-minute 95th-percentile figure while contributing completely differently to the pooled rank, because the pooled rank depends on how many samples sit above a given value, not on where one interval's own cut happened to land. A stored scalar no longer carries that. Weighting the average by each interval's request count fixes the first problem and not the second. A count-weighted mean of quantiles is still a mean, and a mean of quantiles is not the quantile of the pooled samples. There is a subtler honesty problem underneath both. When a system degrades it often *completes fewer requests*, so the worst period contributes the fewest samples to the pooled rank. Pooling is the correct arithmetic, but it is not automatically the most alarming arithmetic — which is why the request count per interval belongs beside the figure in any report. ## The one thing the stored per-interval figures still tell you They **bound** the true value, and the bound costs nothing: - No more than 5% of any interval's samples exceed that interval's own 95th percentile. So no more than 5% of the pooled samples exceed the **largest** of the per-interval figures: the run's true 95th percentile is at most that maximum. - At the **smallest** of the per-interval figures, every interval has at most 95% of its own samples at or below it, so the pooled fraction at or below it is at most 95%: the run's true 95th percentile is at least that minimum. - Both bounds hold whatever the per-interval sample counts were, so they survive a wildly uneven run. The honest statement available from stored per-interval quantiles alone is therefore an interval — *"the run's 95th percentile lies somewhere between 420 ms and 1.9 s"* — and an average is one arbitrary point inside that interval with no property that recommends it over any other. ## What has to have been retained What the result store keeps decides what a merged run report can honestly compute, and the decision cannot be revisited once the run is over. | What was retained | What a whole-run figure can be | Cost | |---|---|---| | Every request's time | Any percentile, exactly; any regrouping by interval, operation or outcome | Grows with request count; by far the largest | | Counts per latency bucket, per interval | Any percentile, to within one bucket's width; counts add bucket by bucket | Small and roughly fixed per interval | | One quantile figure per interval | Only the minimum-to-maximum bound above; no merge is possible | Smallest, and it forecloses the question | A store that keeps only pre-aggregated per-interval summaries can draw an excellent trend over time and cannot answer *"what was this run's 95th percentile"* at all. A store that keeps raw per-request samples answers every regrouping question anyone thinks of afterwards, and pays for it in volume and in retention policy. A mergeable structure of bucket counts sits between the two, which is where long runs usually settle: the merge is a bucket-by-bucket addition, and the price is a bounded resolution error rather than a lost question. ## What to do - Decide retention **before** the run, working backwards from the figures the report will have to carry. - Merge raw samples or bucket counts; never merge stored quantiles. - If only per-interval quantiles survive, publish the bound and label it as a bound. - Keep the per-interval series as a series — a shape over time is what it is genuinely good for — and never collapse it into one number by averaging.