How would you define a latency SLI for an HTTP API, and why is "average response time under 300 ms" the wrong shape for one?

## State it as a ratio, not as a statistic The first correction is structural. A latency SLI is written as **the proportion of valid requests that completed faster than a threshold**, over a window — for example, *99% of checkout requests completed under 400 ms over the trailing 28 days*. It is not written as "the 99th percentile is under 400 ms", even though the two sound equivalent, and it is certainly not written as an average. Stating it as a ratio matters for two practical reasons. First, **percentiles do not aggregate.** You cannot average the p99 of ten servers to get the fleet p99, and you cannot average this hour's p99 with last hour's to get the two-hour p99. A percentile is a quantile of a distribution; combining them requires the underlying distribution, normally the histogram buckets. A ratio of counts, by contrast, adds up perfectly across servers, shards, regions and time windows — you sum the fast requests and sum the total. Any team that reports "average p99 across our instances" has produced a number with no defined meaning. Second, the ratio form composes with everything above it: the same shape of good-events-over-valid-events used for availability, so one machinery serves both. ## Why the average is the wrong statistic Latency distributions in real systems are long-tailed and usually multi-modal. A read path with a cache produces a mode at a few milliseconds (hits) and another at hundreds of milliseconds or seconds (misses, cold shards, retries, GC pauses, connection re-establishment). The mean falls in the empty valley between them — a value that describes no actual request. Worse, the mean is insensitive exactly where you need sensitivity. If one request in a thousand takes 30 seconds and the rest take 20 ms, the mean is about 50 ms and looks fine, while 0.1% of your users are staring at a spinner. Scaled to a service handling ten million requests a day, that is ten thousand abandoned sessions a day sitting invisibly inside a healthy-looking average. The tail is not noise; it is a population of users. ## Choosing the threshold The threshold is the substantive decision, and "300 ms because it sounds good" is the weak answer. Better sources, roughly in order of strength: - **What the caller can absorb.** If a page cannot render until this call returns and the product's budget for that page is one second, and two other calls sit in the same chain, your share is a few hundred milliseconds. Latency thresholds should be *derived from a budget across the chain*, not chosen per service in isolation. - **The knee of the observed curve.** Plot latency against a behavioural outcome — abandonment, retry rate, conversion. There is usually a region where the outcome is flat and then bends sharply. Setting the threshold at that knee makes the indicator track something real. - **Current achievable performance.** If the service currently serves 99.5% of requests under 250 ms, a 400 ms threshold is honest and leaves room; a 100 ms threshold means you have committed to a rewrite. A threshold that the service already meets 99.999% of the time is a vanity indicator — it will never move, and you will learn nothing from it. ## Choosing the percentile The percentile is a statement about *whose* experience you are protecting. - The **50th** describes the typical user and is nearly useless as a guarantee — half your traffic is allowed to be arbitrarily slow. - The **95th/99th** protect the tail. This is where the users on cold caches, large accounts, distant regions and unlucky retries live. In many businesses the heaviest users have the largest payloads and therefore live permanently in your tail — the 99th percentile is disproportionately your biggest customers. - The **99.9th and beyond** get expensive fast and increasingly measure your own infrastructure's noise floor rather than your code. At low traffic they are also statistically meaningless: with 1,000 requests an hour, the 99.9th percentile is a single request. A common, defensible pattern is a **two-threshold** indicator: a high proportion under a comfortable bound and a smaller proportion under a tight one — for example 99% under 400 ms *and* 90% under 100 ms. That captures both "nobody waits absurdly long" and "the normal case feels instant" without pretending one number does both. ## The fast-failure trap A latency indicator computed over *all* responses will report beautifully during a total outage, because a service returning HTTP 500 from its front door does so in single-digit milliseconds. The fix is to define the valid-event set explicitly: count only responses that were also successful, or count errors as failures of the latency indicator too. Whichever you choose, choose deliberately, because the default of "all responses, any status" is actively misleading in exactly the moments you care about. ``` good = requests where status is successful AND duration <= threshold valid = requests where status is successful (or all real user requests) SLI = good / valid ``` ## Where the number comes from Finally, latency has to be measured at the same vantage point as your other indicators, and it must include the time the request spent queued before your code ran. An in-process timer that starts when a handler is entered misses the time the request spent waiting in the accept queue during overload — which is precisely the time the user experienced. Timing at the ingress or from the client is what makes queueing visible.