Alerting Philosophy

## The four signals The golden signals are a checklist for instrumenting any request-serving system, published in Google's *Site Reliability Engineering* (2016) in the monitoring chapter. They are useful here as the **symptom taxonomy** — the vocabulary you use to decide what deserves a page. **Latency** — how long a request takes. The important discipline is to read it as a distribution, not an average, and to separate successful from failed requests. A fast 500 will drag your average latency *down* and make an outage look like a performance improvement, so error responses must be excluded or measured separately. **Traffic** — the demand placed on the system, in whatever unit is natural: HTTP requests per second, sessions, messages consumed per second, bytes served. It is the denominator for the other signals and the context that explains them. **Errors** — the rate of requests that failed. Failure is broader than a 5xx status. It includes responses that were technically successful but wrong, responses that arrived after the caller had already timed out, and policy failures such as a request served without the personalised content it was supposed to carry. Deciding what counts as an error for a given service is most of the work. **Saturation** — how full the service's most constrained resource is. For one service that is memory, for another it is IOPS, for another a fixed worker pool or a database connection limit. Saturation matters because most systems degrade sharply once a resource approaches full, well before it hits 100%. ## Which of these page Apply the symptom test — *would a user feel this?* — to each signal. - **Errors: page.** A failed request is user harm by definition. An alert on the ratio of failed to total requests over a short window is the single most valuable alert most services have. - **Latency: page.** Slow enough is indistinguishable from broken. Alert on a tail percentile — 95th or 99th — against a threshold derived from what the caller can tolerate, not from what the service currently does. - **Traffic: usually not, with one exception.** Rising traffic is not a fault; it is the reason you have capacity planning. But traffic falling to *zero*, or far below the value the same weekday and hour normally shows, is a symptom: it usually means a load balancer, DNS entry, upstream gateway or client release has stopped requests from arriving. A no-traffic alert also protects you from the failure mode where an error-ratio alert goes quiet simply because nothing is being served. - **Saturation: no — ticket it.** Saturation is machine state. A thread pool at 100% for ten minutes with healthy latency and no errors means the service is sized close to its demand, which is a planning conversation. The same pool at 100% *with* rising latency is already showing up in the latency signal, and that is the signal that should carry the page. ## Two related checklists Interviewers sometimes ask how the golden signals relate to other mnemonics. The RED method — rate, errors, duration — is the request-oriented subset, essentially traffic, errors and latency: it is what you apply to a *service*. The USE method — utilisation, saturation, errors — is resource-oriented and is what you apply to a *resource* such as a disk, a NIC or a CPU. The mapping is clean: RED is your paging vocabulary, USE is your diagnosis vocabulary, and the golden signals span both because they include saturation. ## Defining them for a real service The checklist is worth little until it is instantiated. For a checkout API the four might be: latency = 99th percentile of successful POST /checkout calls measured at the edge; traffic = checkout attempts per second; errors = the fraction of attempts returning 5xx or failing payment authorisation for infrastructure reasons; saturation = payment-provider connection pool occupancy plus database CPU. Notice that three of those definitions required a decision about what counts. That is the part that separates a candidate reciting four words from one who has instrumented a service. A common trap is to define errors as "5xx responses" and miss the outage where the service returned a cheerful 200 with an empty cart. ## Non-request systems The signals need translation for a pipeline or a consumer. Latency becomes end-to-end freshness or the age of the oldest unprocessed item. Traffic becomes items ingested per second. Errors becomes the fraction of items that failed or were dead-lettered. Saturation becomes consumer lag against the partition's write rate. The symptom test survives translation intact: freshness and failed items are what a downstream consumer feels; lag on its own is not, until it stops draining.