In PromQL, how do you compute a service-wide 95th percentile from histogram bucket series, and what does histogram_quantile assume?

## The inputs the function needs A Prometheus classic histogram is read as a family of cumulative bucket series, each carrying a label `le` — "less than or equal to" — whose value is that bucket's upper bound, plus an `le="+Inf"` series counting every observation. `histogram_quantile` takes a quantile between 0 and 1 and an instant vector of those bucket series, and returns an estimate of the value at that quantile. Everything that goes wrong with the function goes wrong in what you hand it as its second argument. ## The query, built from the inside out For a hearing-search endpoint whose latency histogram is `hearing_search_duration_seconds_bucket`, the service-wide 95th percentile over the last five minutes is: ```promql histogram_quantile(0.95, sum by (le) (rate(hearing_search_duration_seconds_bucket[5m]))) ``` Read it inside out, because each layer does something the next one requires. 1. **`rate(..._bucket[5m])`** — bucket series are counters. Without a rate, or an `increase`, you are asking for the quantile over the entire lifetime of every process since it started, a figure that drifts glacially and never reflects the last five minutes. 2. **`sum by (le) (...)`** — this is the aggregation across replicas, and `le` is the one label that must survive it. Buckets are additive along the `le` dimension: adding the `le="0.25"` counts from 41 replicas gives the fleet-wide number of requests served under 250 ms. Aggregate `le` away and the histogram is destroyed; the function then has nothing to interpolate over. 3. **`histogram_quantile(0.95, ...)`** — finds the bucket the 95th percentile falls into and interpolates within it. If you want the percentile broken out per courtroom, the grouping becomes `sum by (le, courtroom)`. `le` is always in the list. ## What the function assumes - **Observations are spread evenly inside each bucket.** The function locates the bucket containing the quantile and interpolates linearly between that bucket's lower and upper bounds. Real latencies are not uniform inside a bucket, so the answer is an estimate, not a measurement. - **The bucket boundaries are identical across everything you aggregated.** Two services with different bucket layouts summed by `le` produce a set of series that is not a coherent histogram. The output looks perfectly normal and means nothing, which makes this the most dangerous failure on this function. - **The `+Inf` bucket is present**, because that is where the total observation count comes from. - **The lowest bucket starts at zero**, which is the usual case for durations and sizes. ## How it degrades | Situation | What you see | |---|---| | Quantile falls above the highest finite bound | the highest finite bound, flat and unchanging | | One bucket holds most observations | a value pinned near that bucket's bounds, insensitive to real change | | Too few boundaries around the real latency | large quantised jumps as traffic shifts between buckets | | No observations in the window | no result rather than zero | The first row is the one to recognise on sight. A p99 graph sitting perfectly flat on a round number such as 10 is almost never a real latency plateau; it means every slow request landed in the unbounded top bucket, and since the function cannot interpolate to infinity it reports the last boundary it knows. The fix is on the instrumentation side, choosing boundaries that bracket the latencies you actually care about, not in the query. ## Why you cannot average percentiles A tempting query is an average of per-replica percentiles: compute the estimate for each replica, then `avg` them. It is wrong, and not by a rounding error. A percentile is a position in a distribution, not a quantity. The 95th percentile of the combined traffic of 41 replicas is not the average of their individual 95th percentiles, and the error grows with how unevenly the replicas are loaded: a replica serving 12 requests with a slow tail contributes exactly as much to the average as one serving 4,800 fast ones. The composition rule is: **aggregate the buckets, then estimate; never estimate, then aggregate.** The same rule explains why a quantile that arrives as a plain pre-computed number, calculated inside the application before export, cannot be re-aggregated at all. The distribution it came from is gone, and no query can reconstruct it. - Correct: `histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m])))` - Wrong: an `avg` wrapped around a per-replica `histogram_quantile` - Wrong: `histogram_quantile(0.95, sum(rate(x_bucket[5m])))`, which aggregates `le` away