A request crosses four services, each with a known 99th-percentile latency. Why is the end-to-end 99th percentile not their sum?

seniorshould knowfreq 44%

answer

Tails do not line up
It prices every hop at its worst
The free bound is a weaker rank
Correlated hops break the independence story
Time the whole path per request

basics

~20 s

Summing per-hop 99th percentiles prices a request in which all four hops were simultaneously in their own slowest one percent, which is rare — so the sum usually overstates. Correlated hops can make it understate. Measure whole-path time per request instead.

solid answer

~50 s

A percentile describes one distribution; it is not a budget line that adds. Adding four hops' 99th-percentile times prices the request in which all four were simultaneously in their own slowest one percent — under independence a vanishingly rare request, so the sum sits far above the real end-to-end figure. The bound you actually get is weaker: the chance that any of four hops exceeds its own 99th percentile is at most 4%, so the sum bounds the 96th percentile of the total, not the 99th. In the other direction, hops share resources and go slow together, and that correlation fattens the joint slow event beyond what independence predicts — so the error is not even signed. Averages add along a path; percentiles never do. The honest figure comes from timing each request across the whole path and ranking those totals.

A percentile describes one distribution; it is not a budget line that adds. Adding four hops' 99th-percentile times prices the request in which **all four were simultaneously in their own slowest one percent** — under independence a vanishingly rare request, so the sum sits far above the real end-to-end figure. The bound you actually get is weaker: the chance that *any* of four hops exceeds its own 99th percentile is at most 4%, so the sum bounds the 96th percentile of the total, not the 99th. In the other direction, hops share resources and go slow together, and that correlation fattens the joint slow event beyond what independence predicts — so the error is not even signed. Averages add along a path; percentiles never do. The honest figure comes from timing each request across the whole path and ranking those totals.

code

pseudocode · 18 lines

pseudocode

# WRONG: adding stored per-hop high percentiles
predicted = sum(hop.p99 for hop in hops)

# What the union bound actually licenses, for four hops:
#   P(any hop above its own 99th) <= 4 * 0.01 = 0.04
#   so `predicted` bounds the 96th percentile of the total,
#   not the 99th
planning_ceiling = predicted   # design budget only, never a report

# RIGHT: compose the samples, not the summaries
totals = []
for req in run.requests:
    totals.append(sum(req.hop_time[h] for h in hops))
end_to_end_p99 = value_at_rank(sort(totals), 0.99)

# fan-out is governed by the maximum, not the sum
for req in run.requests:
    req.wait = max(req.call_time[c] for c in req.parallel_calls)

go deeper

for a junior

Recall that a percentile describes one set of measurements. Adding two of them is not the same operation as adding two durations, and an end-to-end time has to be measured end to end.

for a middle

Explain why the sum prices a request in which every hop is simultaneously at its worst, and which quantity does compose along a path: the average always, the variance only when hop times are uncorrelated.

for a senior

Demonstrate that you rank whole-path totals per request, and that you can say in which direction a naive sum is wrong for a given system — including when shared saturation makes several hops go slow on the same requests.

for a principal

Own the latency-budget conversation: how a whole-path target is allocated across hops, why the allocation is deliberately conservative, and where you accept a modelled estimate instead of paying for whole-path measurement everywhere.

What the sum actually prices

Every hop on a request's path has its own distribution of times. A 99th-percentile figure for one hop says: on about one call in a hundred, this hop took at least this long. Adding four such figures produces the time of a request in which all four hops were simultaneously in their own slowest one percent. If hop times were independent, a request like that arrives roughly once in a hundred million. It is not the 99th-percentile request — it is a request that almost certainly never happened during the run at all.

So the sum is a price, and it is the price of the wrong event. The 99th-percentile request end to end is normally one that was moderately unlucky in two or three places at once, and no arithmetic over per-hop summary figures can express that, because the summaries no longer say which request was unlucky where.

The bound you do get for free

There is one rigorous statement available without assuming anything about independence, and it is worth knowing precisely because it is weaker than people expect:

The probability that at least one of the four hops exceeds its own 99th percentile is at most 4% — the sum of the four individual 1% probabilities.
So at least 96% of requests have every hop at or below its own 99th percentile, and therefore a total at or below the sum of the four figures.
The sum of four 99th percentiles therefore bounds the 96th percentile of the end-to-end time, not the 99th.
To bound the end-to-end 99th percentile this way, you would need each hop's 99.75th percentile — a figure that is far noisier, needs far more samples, and that most result stores do not carry.

That is the honest version of "the sum is conservative": it is conservative about a lower rank than the one you were asked about.

The other direction: hops that go slow together

The union bound above needs no independence, but the "vanishingly rare" intuition does, and real hops are correlated. They share a network, a host, a downstream store, a periodic runtime pause, a queue that backs up, a retry surge that hits everything at once. Under positive correlation the joint slow event is far more likely than independence predicts, and the true end-to-end 99th percentile moves towards the sum, occasionally past what an independence-based estimate would allow.

The practical consequence is that the error in a naive sum is not even signed. It is usually a large overstatement, sometimes far closer to the truth than the arithmetic deserves, and never a quantity you can correct with a fixed fudge factor.

What composes and what does not

Quantity	Along a sequence of hops	Condition
The average time	Adds	Always, with no independence needed
The variance	Adds	Only when hop times are uncorrelated
A percentile	Does not add	No condition makes it add
The maximum	Governs a parallel wait, not a sequence	Fan-out, where the slowest call holds the request

The first row is why an average-based latency budget is arithmetically sound and a percentile-based one is not: expectation is linear, so per-hop averages sum to the whole-path average whatever the correlations. Nothing similar is true of a rank.

What per-hop high percentiles are legitimately for

Budget allocation at design time. Split an end-to-end target across hops and give each hop a share. The allocation is conservative by construction — if every hop stays inside its share, the total stays inside the target — and that conservatism is a feature of a planning device, not a defect in a prediction.
Ranking attention. Ordering hops by their own high percentiles tells you where the slow calls live and which hop is worth instrumenting further.
Per-hop alarms. A hop's own distribution is the right thing to watch for a change in that hop, independently of what the whole path is doing.

What they are not for is publishing a predicted end-to-end figure, or comparing a summed number against an agreed whole-path response-time limit.

Getting a real end-to-end figure

Time each request across the whole path, with one start and one finish per request, and record the per-hop times against that same request rather than as separate populations.
Rank the totals and read the percentile off them. That single set of whole-path totals is the only thing an end-to-end percentile can honestly come from.
Expect the composition to surprise you: the request at the whole-path 99th percentile is often at no hop's own 99th percentile.
Fan-out deserves its own note. When a request waits on many parallel calls, the slowest governs, and the chance that at least one of a hundred independent calls lands in its own slowest one percent is about 63%. Behaviour that is negligible per call becomes the median experience once a request depends on enough calls at once.

## What the sum actually prices Every hop on a request's path has its own distribution of times. A 99th-percentile figure for one hop says: on about one call in a hundred, this hop took at least this long. Adding four such figures produces the time of a request in which **all four hops were simultaneously in their own slowest one percent**. If hop times were independent, a request like that arrives roughly once in a hundred million. It is not the 99th-percentile request — it is a request that almost certainly never happened during the run at all. So the sum is a price, and it is the price of the wrong event. The 99th-percentile request end to end is normally one that was *moderately* unlucky in two or three places at once, and no arithmetic over per-hop summary figures can express that, because the summaries no longer say which request was unlucky where. ## The bound you do get for free There is one rigorous statement available without assuming anything about independence, and it is worth knowing precisely because it is weaker than people expect: - The probability that at least one of the four hops exceeds its own 99th percentile is at most 4% — the sum of the four individual 1% probabilities. - So at least 96% of requests have every hop at or below its own 99th percentile, and therefore a total at or below the sum of the four figures. - The sum of four 99th percentiles therefore bounds the **96th** percentile of the end-to-end time, not the 99th. - To bound the end-to-end 99th percentile this way, you would need each hop's 99.75th percentile — a figure that is far noisier, needs far more samples, and that most result stores do not carry. That is the honest version of "the sum is conservative": it is conservative about a lower rank than the one you were asked about. ## The other direction: hops that go slow together The union bound above needs no independence, but the *"vanishingly rare"* intuition does, and real hops are correlated. They share a network, a host, a downstream store, a periodic runtime pause, a queue that backs up, a retry surge that hits everything at once. Under positive correlation the joint slow event is far more likely than independence predicts, and the true end-to-end 99th percentile moves *towards* the sum, occasionally past what an independence-based estimate would allow. The practical consequence is that the error in a naive sum is not even signed. It is usually a large overstatement, sometimes far closer to the truth than the arithmetic deserves, and never a quantity you can correct with a fixed fudge factor. ## What composes and what does not | Quantity | Along a sequence of hops | Condition | |---|---|---| | The average time | Adds | Always, with no independence needed | | The variance | Adds | Only when hop times are uncorrelated | | A percentile | Does not add | No condition makes it add | | The maximum | Governs a parallel wait, not a sequence | Fan-out, where the slowest call holds the request | The first row is why an average-based latency budget is arithmetically sound and a percentile-based one is not: expectation is linear, so per-hop averages sum to the whole-path average whatever the correlations. Nothing similar is true of a rank. ## What per-hop high percentiles are legitimately for 1. **Budget allocation at design time.** Split an end-to-end target across hops and give each hop a share. The allocation is conservative by construction — if every hop stays inside its share, the total stays inside the target — and that conservatism is a feature of a planning device, not a defect in a prediction. 2. **Ranking attention.** Ordering hops by their own high percentiles tells you where the slow calls live and which hop is worth instrumenting further. 3. **Per-hop alarms.** A hop's own distribution is the right thing to watch for a change *in that hop*, independently of what the whole path is doing. What they are not for is publishing a predicted end-to-end figure, or comparing a summed number against an agreed whole-path response-time limit. ## Getting a real end-to-end figure - Time each request across the whole path, with one start and one finish per request, and record the per-hop times against that same request rather than as separate populations. - Rank the totals and read the percentile off them. That single set of whole-path totals is the only thing an end-to-end percentile can honestly come from. - Expect the composition to surprise you: the request at the whole-path 99th percentile is often at no hop's own 99th percentile. - Fan-out deserves its own note. When a request waits on many parallel calls, the slowest governs, and the chance that at least one of a hundred independent calls lands in its own slowest one percent is about 63%. Behaviour that is negligible per call becomes the median experience once a request depends on enough calls at once.

A request fans out to a hundred parallel calls and waits for all of them. How does that change the arithmetic?
A parallel wait is governed by the maximum, not the sum. If each call independently has a one-in-a-hundred chance of landing in its own slowest one percent, the chance that at least one of a hundred does is about 63%. So the median fan-out request already contains a call from that slow one percent, and behaviour that is negligible per call becomes the common experience.
If the sum is not a prediction, how would you set a per-hop target from an end-to-end target?
Allocate the end-to-end target across hops as shares, accepting that the allocation is deliberately conservative because meeting every share guarantees meeting the target. Then verify with measured whole-path times rather than by adding the achieved per-hop figures back up. Allocation is a design device; verification is always a measured distribution of totals.
Which composition assumption does adding variances rely on, and why does it matter here?
Independence. Average times add along a path with no assumption at all, because expectation is linear, but variances add only when hop times are uncorrelated. Once hops share a saturated resource, the variance of the total exceeds the sum of the hop variances, and any estimate built on independent hops is optimistic about how bad the slow end gets.

saying these in an interview costs you the question

Adds per-hop 99th percentiles and reports the total
Assumes hops are independent without ever checking
Claims the sum is always a safe upper bound
Treats averages and percentiles as composing alike
Never times a whole request path end to end

A request crosses four services, each with a known 99th-percentile latency. Why is the end-to-end 99th percentile not their sum?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What the sum actually prices

The bound you do get for free

The other direction: hops that go slow together

What composes and what does not

What per-hop high percentiles are legitimately for

Getting a real end-to-end figure

▾common follow-ups3 questions

▾red flags5 flags

What the sum actually prices

The bound you do get for free

The other direction: hops that go slow together

What composes and what does not

What per-hop high percentiles are legitimately for

Getting a real end-to-end figure

answer

go deeper