A product NFR states an API must respond in under 300ms at p99. The request fans out to a database call, an auth check, and two downstream microservices. How do you turn that single latency number into a design constraint across the call chain?

## Draw the critical path before you divide anything Turning a single end-to-end latency NFR into a workable design means recognizing that a p99 target on a composite operation is a budget to be allocated across a call graph, not a number that applies independently to each hop. The mechanism starts with drawing the actual critical path: for the request described — a DB call, an auth check, and two downstream microservice calls — the first design question is which of these are strictly synchronous and blocking versus which could be made asynchronous, cached, or removed from the request path entirely, because every hop kept on the synchronous critical path directly eats into the 300ms budget, while anything moved off it becomes free from the caller's perspective. ## Allocate at the percentile that matters For what remains on the critical path, the architect allocates a slice of the total budget to each component, but must do so at the percentile that matters, **not at the average**. This is the crux most engineers get wrong: if a request calls three components sequentially and each independently has a p99 latency equal to your allotted 100ms slice, the probability that at least one of the three exceeds its own p99 on any given request is meaningfully higher than 1% — tail latencies compound across a call chain, so the end-to-end p99 of a sequential chain is typically worse than the sum of the individual p50s, once you account for correlated slow periods (e.g., a GC pause or a noisy-neighbor event affecting multiple hops at once). This is why serious latency budgeting reserves generous headroom — often allocating each hop a target notably tighter than its 'fair share' of the total, plus a fixed slop factor (commonly 15-30ms) for serialization, network jitter, and connection setup that's hard to attribute cleanly to any single hop. ## Overlap the calls that can overlap The second major mechanism is exploiting parallelism. If the two downstream microservice calls are independent of each other (neither needs the other's output), calling them concurrently rather than sequentially means their latencies overlap instead of summing — the fan-out's contribution to the budget becomes roughly `max(latency_A, latency_B)` instead of `latency_A + latency_B`. This single change is often the highest-leverage lever available, because it converts an additive latency cost into something close to the slower of the two, for free (aside from added implementation and error-handling complexity from managing concurrent calls). ## Who owns the sum Why this discipline exists: business and product NFRs are stated as a single end-to-end number because that's what the user experiences, but engineering has to operate on the decomposed pieces, since each hop is owned, deployed, and scaled independently, often by different teams. Without an explicit budget allocation, teams optimize locally (each service owner is proud their own p99 is fast) while the composite user-facing latency silently drifts past the NFR because no one owns the sum. Latency budgeting is the practice of making that ownership explicit and quantitative before the system is built, not discovering the problem in production. ## What a tight budget costs The trade-offs are real. Tight per-hop budgets force architectural choices with cost: - **Aggressive timeouts** on downstream calls mean you must design a fallback (cached/stale data, degraded response, or a documented partial-failure behavior) for when a call is cut off, which is more implementation complexity than 'just wait for it.' - **Caching** a downstream result to remove it from the critical path introduces staleness the product has to accept. - **Parallelizing** calls that were sequential for a reason (e.g., one call's output feeds the other) may not be possible at all, forcing a harder architectural change like denormalizing data so the second call doesn't need the first's result. - **Reserving generous tail-latency slack** means the 'average case' budget is much smaller than 300ms — a system genuinely built to hit 300ms at p99 might run at 80-120ms at p50, which can look like wasted headroom to someone reading dashboards without understanding why the margin exists. ## How it fails in production Failure modes in production are recognizable: 1. A team validates the budget using average (p50) latencies per hop, sums them, declares the design compliant, and then the real p99 blows through the target because tail latencies weren't modeled. 2. A synchronous call chain has no timeout on one downstream dependency, so that dependency's own p99 degradation directly becomes the caller's p99 (and eventually p100) failure. 3. Or a 'fast enough on average' downstream service has an occasional multi-second GC-pause tail that, called synchronously without a timeout, occasionally blows the entire request's budget even though it barely shows up in that service's own p50/p99 dashboards. ## A worked allocation against the 300ms target A worked example: for the 300ms p99 API described, an architect might allocate: | Hop on the critical path | Slice | |---|---| | network/gateway overhead | ~30ms | | auth check | ~20ms, backed by a fast in-memory token cache, not a synchronous call to a remote auth service | | the DB query | ~80ms, with an index review to keep p99 tight | | the two downstream calls | fanned out concurrently with a hard 120ms timeout each and a fallback of cached/default data on timeout — bringing the fan-out's contribution to ~120ms instead of up to 240ms sequential | Summed with margin, that's roughly 250ms allocated against the 300ms budget, leaving ~50ms of slack for jitter and retries, validated afterward with end-to-end load testing measuring actual p99, not the sum of component averages.