Service Levels & Error Budgets

An error budget is one minus the SLO target over the window. A 99.9% target across 30 days allows about 43 minutes of unavailability, or 0.1% of requests, before the objective is missed. Spending it means shipping change against that allowance.

An SLI is a measured number, usually good events divided by valid events. An SLO is the internal target that number must meet over a defined window. An SLA is the customer-facing contract, set looser than the SLO, and it is the only one carrying penalties.

Burn rate expresses error-budget consumption as a multiple of the sustainable pace: rate 1 spends the whole budget exactly at the window's end. Compute it as the observed bad-event ratio divided by (1 minus the SLO target).

Define latency as the proportion of valid requests served faster than an explicit threshold — for example 99% of checkout requests under 400 ms. An average hides the slow tail entirely: a handful of thirty-second requests barely move it while those users are the ones leaving.

A 100% target is unachievable and forbids all change, since every deploy and dependency can fail. Choose between 99.9% and 99.99% by what users can actually perceive and what the extra nine costs: 99.9% allows about 43 minutes of downtime a month, 99.99% about 4.

Use two paging tiers: 14.4x averaged over 1 hour for fast burns and 6x over 6 hours for slower ones, each ANDed with a short window (5 minutes and 30 minutes) of the same threshold. The long window sets sensitivity; the short one proves the burn is still happening.

A budget policy, agreed in advance by engineering, SRE and product, states what happens at zero: feature releases stop while reliability and security fixes continue, reliability work is funded, and a single named owner may override — with every override logged and counted.

A static error-ratio threshold ignores both the SLO and how long the condition lasts. It pages too early on brief or low-volume blips that cost almost no budget, and stays silent through a sustained sub-threshold error rate that quietly drains the entire budget in days.

Default to 5xx as bad and 2xx/3xx as good, then decide each grey case deliberately: health checks and scanner traffic come out of the denominator, 429s count as bad when you shed because you were overloaded, and 4xx stays out unless your own change caused it.

Time-based budgets count bad minutes and weight every minute equally; request-based budgets count failed requests and weight an incident by the traffic it hit. Partial outages and off-peak incidents cost very differently under the two.

A rolling window recomputes continuously, so an outage ages out gradually and there is never a reset to wait for. A calendar window aligns with billing and planning cycles but resets abruptly, so an early outage poisons the whole period and a late one barely registers.

The denominator decides what the service is accountable for. Health checks and bot traffic inflate it with easy successes that mask real user failures, and quietly excluding awkward traffic makes the indicator look better without the service improving.

The outage stays inside the trailing one-hour average until it rolls out of the window, so the condition remains true long after the burn stops. The fix is to AND the long window with a short one, typically a twelfth of its length, so the alert clears roughly five minutes after recovery.

Each vantage point sees a different slice of the request path. Application counters miss everything that never reached the process, load-balancer logs miss DNS and network failures, client telemetry only reports from clients healthy enough to report, and probes measure a synthetic journey at low resolution.

Define indicators on the data, not the job: freshness (how recently the output was updated), coverage or completeness (share of input records processed), and correctness (share of outputs that validate). A zero-exit job can emit nothing at all, so exit status says nothing about what a consumer received.

A permanently unspent error budget is not a win. It usually means the target is looser than what users notice, or the team is buying reliability nobody asked for. Tighten the SLO, or deliberately spend the budget on velocity.

Run a tighter internal target than the contract — commonly 99.95% behind a 99.9% SLA. The gap is the team's reaction margin: internal alarms fire and remediation starts while the customer is still well inside the promised level and owed nothing.

Hold the budget-spend policy fixed, not the multipliers. Burn rate already normalises across different SLO targets, but the multipliers are derived from the window length, so a 7-day SLO needs 3.36x where a 30-day SLO needs 14.4x, and low-traffic services need an extra event-count guard.

Standardise the shape and the measurement point, not the targets: compute one default availability and latency indicator centrally from a shared ingress for every service, and let teams add journey-specific indicators — freshness, correctness — that only they can define.

Count it. An error budget measures user experience, not team fault, and exempting dependency failures hides the case for removing the dependency. Adjust the policy response to the cause instead of adjusting the measurement.

Start from the worst measured month, not the average, then compose the dependency chain: serial dependencies multiply, so three components at 99.99% cap you near 99.97%. Four nines allows 4.32 minutes a month, which is an automated-failover architecture, not a promise you can talk your way into.