A Kubernetes-deployed microservice exposes both a liveness probe and a readiness probe, plus a /metrics endpoint scraped by Prometheus. What's the functional difference between the two probes, and what happens to the service's traffic and pod status if only one of them is misconfigured to always report healthy?

## Health checks and what each probe answers Health checks and metrics aggregation serve two different, complementary jobs in keeping a fleet of microservice instances operating correctly, and conflating them is a common source of production incidents. - A **liveness probe** answers a narrow question about a single instance: "is this process in a state where restarting it is likely to help?" It's typically a lightweight endpoint (or even just a TCP connect check) that an orchestrator like Kubernetes polls on an interval; if it fails enough consecutive times, the orchestrator kills and restarts that specific container, on the assumption that whatever's wrong (a deadlock, a memory leak that's degraded the process beyond recovery, an unresponsive event loop) is best fixed by starting fresh. - A **readiness probe** answers a different question: "should this specific instance currently be sent new traffic?" Failing readiness doesn't kill or restart anything - it simply removes that pod's IP from the set of endpoints the load balancer/service routes to, so no new requests reach it, while existing in-flight requests can still finish and the process itself keeps running. This distinction exists because there are states an instance can legitimately be in where it needs to stop receiving new work without needing to be killed: still warming up a cache after startup, temporarily unable to reach a downstream dependency it needs, or intentionally draining before a planned shutdown. Many orchestrators also support a third probe, a **startup probe**, specifically to give a slow-booting service a generous grace period before liveness checks even begin being evaluated, since without it a service that legitimately takes 60 seconds to load a large in-memory index would otherwise get killed repeatedly by an impatient liveness probe before it ever finishes starting. ## What metrics aggregation adds Metrics aggregation operates at a different level entirely: instead of a binary per-instance signal, it collects continuous numeric time series - request counts, error counts, latency histograms - from every instance and rolls them up centrally, most commonly via **Prometheus**, which pulls (scrapes) a `/metrics` endpoint on each instance at a regular interval and stores the resulting time series, queryable and alertable via PromQL and typically visualized in Grafana. A widely used framework for what to actually measure is the **RED method** from Google's SRE practice: - **Rate** (requests per second), - **Errors** (rate of failing requests), - and **Duration** (latency distribution, usually as percentiles like p50/p95/p99). Aggregated across the whole fleet and over time, RED metrics reveal gradual degradation trends - a creeping rise in p99 latency, an error rate ticking up on one specific endpoint - well before enough individual instances would actually be unhealthy enough to fail a binary health check; this is precisely the gap probes can't fill, because a probe only tells you about one instance's current pass/fail state, not about trends or about problems that are real but not yet severe enough to flip that binary signal. ## When the mechanism causes the outage The trade-off and failure-mode space here is significant because these mechanisms sit directly in the availability control loop and misconfiguring them actively causes outages rather than merely failing to prevent them. 1. **Readiness that always reports healthy.** If a readiness probe is misconfigured to always report healthy regardless of the instance's real internal state - a common mistake is hardcoding it to return 200 instead of actually checking a dependency like the database connection - an instance that's genuinely failing (lost its DB connection, exhausted a resource pool) stays fully in the load balancer's rotation and keeps receiving its normal share of new traffic, which it then fails to serve, producing errors for real users instead of being quietly drained out to recover. 2. **Liveness too aggressive.** Conversely, a liveness probe that's too aggressive - too short a timeout, or one that depends on a slow downstream call like a database query as part of its own check - can cause what's often called a **restart storm**: during a genuine traffic spike, if response latency including the database's rises fleet-wide, the liveness probe can start failing across many or most instances simultaneously purely because of load, not because those processes are actually stuck. The orchestrator then restarts a large fraction of the fleet at once, which sheds real serving capacity and adds cold-start cost (re-establishing connections, warming caches) across the remaining and restarting instances right as the fleet is already under pressure - the health-checking mechanism itself becomes the proximate cause of a worse outage, a self-inflicted denial of service triggered by the orchestrator's own automation. ## The cardinality pitfall A second operational pitfall specific to metrics is **cardinality**: each unique combination of label values on a metric (e.g., endpoint, status code, and a per-user or per-request identifier) creates a distinct time series the backend must store and index. Using a high-cardinality value as a label - a raw user ID or request ID, rather than a bounded set like endpoint name or status code class - can generate millions of near-permanent time series, which is a well-documented way to blow up storage and query cost or badly degrade a Prometheus deployment, entirely independent of whether the underlying monitoring intent was reasonable. ## Different tools for different jobs A well-designed operational setup treats probes and metrics as different tools for different jobs: - **liveness** is a narrow, dependency-free self-check meant to catch a genuinely stuck process; - **readiness** gates traffic based on real internal state, including downstream dependency health, without being so strict it flaps under normal load; - and fleet-wide **RED metrics** with alerting catch the gradual degradation trends that individual binary checks are structurally unable to see.