When you design a service's health-check endpoint, should it verify that its database and downstream APIs are reachable, or just confirm the process itself is running? What are the trade-offs of 'deep' versus 'shallow' checks?

## Shallow and deep, defined A **shallow health check** verifies only that the process itself can respond — the HTTP server thread accepts a connection and returns a fixed 200, with no interaction beyond the request handler itself. A **deep health check** actively exercises the service's real capabilities: - it pings its **database connection pool**; - calls a **downstream API**; - checks **disk space** or **queue depth**; - and aggregates the results into a composite status, often returning per-dependency detail in the response body rather than a bare pass/fail. The two answer very different questions: | Check | The question it settles | |---|---| | Shallow | shallow answers 'is the process alive,' | | Deep | deep answers 'can this instance actually do useful work right now.' | ## Why deep checks exist Deep checks exist because a process can be technically running while functionally broken. The HTTP server thread can accept connections and return 200 on a shallow check while the connection pool to the database is exhausted, or a required downstream auth service is down, meaning every real request the instance receives will fail. A shallow check would report healthy the entire time, so nothing in the traffic-routing layer knows to avoid that instance, and real users keep hitting an instance that can't help them. Deep checks close that gap by testing actual capability rather than mere existence. ## The trade-offs The trade-offs run in both directions. - **Deep checks cost load**: if a probe runs every few seconds across dozens or hundreds of instances, and each check makes a fresh call to a shared dependency, the aggregate polling traffic against that dependency can itself become significant, sometimes enough to be the thing that tips an already-stressed database over. - **They also introduce correlated-failure risk**: if a shared downstream dependency degrades, every instance's deep check can fail at nearly the same moment, and depending on which probe carries the deep check, that means either a synchronized crash-loop restart storm (liveness) or the entire fleet marked not-ready simultaneously (readiness) — total outage instead of the intended graceful degradation. - **Shallow checks avoid both costs but pay in false positives**: the load balancer keeps sending traffic to an instance that looks fine but can't actually complete requests, hurting real users while every health dashboard stays green and alerting stays silent. ## The pattern that resolves it The pattern that resolves this in practice is to match depth to purpose and to decouple the check's cost from its frequency. 1. **Liveness should stay shallow** — process-only — always, because its remedy (restart) can never fix an external dependency problem. 2. **Readiness can be selectively deep**, but the dependency check itself should run on a background timer independent of probe frequency, with the readiness endpoint simply returning the cached result instantly; this keeps the probe response fast and cheap while the actual dependency polling happens at a much lower rate, often paired with a circuit breaker so a hanging dependency doesn't pile up slow calls. 3. Where possible, **health status should be modeled as more than binary** — a 'degraded' state for a non-critical dependency lets an instance keep serving the majority of its traffic while a specific feature fails gracefully, rather than pulling the whole instance out of rotation over one non-essential dependency. ## A realistic failure mode, and the fix A realistic production failure mode illustrating why this matters: a fleet of application pods runs a liveness probe that executes a live SQL query against a shared primary database on every check. A brief database failover event, lasting well under two minutes, causes every pod's liveness probe to fail at nearly the same moment; the orchestrator restarts the whole fleet, and because the database is still failing over when the pods come back up, many hit `CrashLoopBackOff` and fail liveness again. The database recovers in under two minutes, but the fleet-wide outage, compounded by exponential restart backoff, lasts substantially longer than the actual database event that caused it. The fix teams apply after such an incident is exactly the pattern above: - move the SQL check out of liveness entirely; - keep liveness as a pure process check; - and put the database check behind a cached, background-refreshed readiness signal so a shared dependency blip degrades traffic routing gracefully instead of triggering a synchronized restart storm.