A team monitoring a fleet of stateless API services builds dashboards using the RED method, while the team monitoring the underlying VMs and disks builds dashboards using the USE method. What does each method measure, and why is one better suited to request-driven services and the other to resources?

## Two ways to pick what to measure RED and USE are two complementary methodologies for choosing which metrics matter, both created to answer a simpler underlying question: out of the effectively infinite number of things you could measure about a system, which handful will actually tell you it's unhealthy? | | RED | USE | |---|---|---| | Target | anything that serves requests: a service, an endpoint, a queue consumer | a finite resource with a hard capacity ceiling, such as a CPU core, a disk, a network interface, or a memory pool | | Signals | Rate, Errors, Duration | Utilization, Saturation, Errors | ## RED, at the service layer RED, popularized around monitoring Kubernetes microservices with Prometheus, targets anything that serves requests: a service, an endpoint, a queue consumer. It defines exactly three signals. - **Rate** is the number of requests the component handles per second, which establishes the baseline traffic context, a latency spike means something different at 10 requests per second than at 10,000. - **Errors** is the rate of requests that failed, typically defined by a protocol-level signal such as an HTTP 5xx status or a gRPC error code, expressed as a fraction or rate rather than a raw count so it's comparable across different traffic levels. - **Duration** is the distribution of how long requests took, almost always reported as percentiles such as p50, p95, and p99 rather than an average, because averages hide the long tail of slow requests that actually hurts users. Together, these three answer whether the service is serving traffic successfully and quickly, which is precisely the question a caller or an SLO cares about. ## USE, at the resource layer USE targets a different kind of thing: a finite resource with a hard capacity ceiling, such as a CPU core, a disk, a network interface, or a memory pool. It also defines exactly three signals, but with a different meaning. - **Utilization** is the percentage of time the resource was busy doing work, for example the percent of time a CPU was not idle. - **Saturation** is the amount of extra work the resource couldn't get to immediately, typically measured as a queue length such as the CPU run queue or the disk I/O queue, and it is the more important of the two for catching trouble early, because a resource can be at high utilization with no queuing, which is fine, or hit saturation well before 100 percent utilization on components with variable service times. - **Errors**, in this framing, means literal hardware or driver-level error counts, like disk I/O errors, distinct from RED's application-level request errors. ## Why you need both The reason both exist, rather than one method covering everything, is that they answer questions at different layers of the stack, and a problem at one layer often doesn't show up cleanly at the other, at least not immediately. A service's RED dashboard can look completely green, low error rate, healthy p99, right up until the moment the underlying VM's CPU saturates and a queue starts building, at which point RED's duration metric will start climbing, but by then the USE dashboard for that VM would already have shown rising saturation minutes earlier as an early warning. Conversely, a disk can sit at 40 percent utilization, which sounds fine, while still causing real user-facing latency if that 40 percent happens to include long, bursty writes that saturate its queue; USE metrics alone won't tell you that this is actually hurting checkout latency, only the RED dashboard for the checkout service will show that. ## The trade-off The trade-off is mostly about where you point which method rather than one method being strictly better. RED requires the component to be something that meaningfully handles requests, so it doesn't apply cleanly to a raw resource like a disk. USE requires the thing being measured to be a finite, quantifiable resource, so it doesn't map cleanly onto an abstract service's business logic, since utilization of a stateless API process isn't a well-defined single number the way CPU utilization is. Both methods deliberately keep the metric count small, three signals each, as a trade-off against completeness: this is by design, to keep dashboards actionable during an incident rather than an overwhelming wall of graphs, at the cost of not directly capturing things neither method was designed for, like business-level correctness or data quality. ## Where dashboards go wrong Common failure modes: - Using RED's Duration as an average instead of percentiles hides exactly the slow-tail requests SLOs are meant to protect. - Skipping the Rate signal on a RED dashboard means an error-rate spike caused by a traffic collapse, fewer total requests but a higher fraction failing, looks identical to a genuine failure spike, when the underlying cause is completely different. - On the USE side, watching Utilization alone and ignoring Saturation is the single most common miss, since a resource can look not that busy by utilization while already queuing work and hurting latency. ## How the pair is used in practice A concrete, real-world pairing: a team running services on Kubernetes typically builds RED dashboards per HTTP endpoint from Prometheus histograms generated by request middleware, and USE dashboards per node from node exporter metrics covering CPU, memory, and disk I/O, with the RED dashboards used to judge whether an SLO is at risk and the USE dashboards used to diagnose why.