How do active and passive health checks work in a load balancer, and what production failure modes can occur if health-check thresholds are configured too aggressively or too loosely?

## Why health checking exists A load balancer can only route around a broken backend if it knows the backend is broken, and health checking is the mechanism that supplies that knowledge. There are two broad families. | Family | Where the signal comes from | |---|---| | **active checks** | the load balancer itself originates synthetic probe traffic | | **passive checks** | the load balancer infers health by observing the outcome of real client requests it is already forwarding | ## Active checks: synthetic probes on an interval An active health check works by the load balancer opening a connection (for L4) or sending an HTTP request (for L7, typically a GET to a dedicated endpoint like `/health` or `/healthz`) to each backend on a configured interval - say every 5 or 10 seconds. The response is judged against expected criteria: - **for TCP checks**, simply whether the connection succeeds; - **for HTTP checks**, whether the status code falls in an expected range (usually 200) within a timeout, and sometimes whether the response body matches an expected pattern. Crucially, the load balancer does not act on a single failed probe - it applies: - an **unhealthy threshold**, a count of consecutive failures (commonly 2-3) required before the backend is pulled out of rotation; - a **healthy threshold**, a count of consecutive successes required before it is added back. This debouncing exists because a single missed probe is common and often meaningless - a momentary network blip, a garbage-collection pause, or a probe that happened to race with a deploy - and pulling a backend out over one failed check would cause unnecessary churn. The health endpoint itself is a design decision with real consequences: - a **shallow check** that just returns 200 from the web server tells you the process is alive but nothing about whether it can reach its database or downstream dependencies; - a **deep check** that verifies DB connectivity and dependency health gives a much more accurate picture of true readiness - at the cost of the health endpoint itself becoming a potential bottleneck or single point of failure if it, say, does an expensive query on every probe. ## Passive checks: riding on real traffic A passive health check instead rides on real traffic: the load balancer watches the actual responses it gets back from each backend as it forwards genuine client requests, and tracks a rolling count or rate of failures - HTTP 5xx statuses, connection resets, timeouts. If that rate crosses a configured threshold within a time window (for example, more than 50% of the last 10 requests failed, or 5 consecutive timeouts), the backend is marked unhealthy and removed from rotation, usually for a cooldown period after which it is retried. This is sometimes called an 'outlier detection' or 'circuit breaker' pattern at the load-balancer level, and it has the advantage of adding zero synthetic load and reflecting the exact failure modes real users are experiencing, including ones a shallow active probe would never surface (e.g. a backend that answers health checks fine but fails specifically on a particular heavy query pattern). Its weakness is that it requires actual traffic to detect a problem at all - a backend with no incoming requests yet, or one that is broken for a code path nobody has hit recently, will look healthy - and by definition some real user requests must fail before the system reacts, unlike an active check which can catch a dead process before any user ever reaches it. ## Thresholds too aggressive: flapping and cascade The failure modes show up at both extremes of threshold tuning. Configuring checks too aggressively - a short interval combined with a low unhealthy threshold, such as checking every 2 seconds and pulling a backend after a single failure - makes the system hypersensitive to transient noise: a backend experiencing a normal JVM garbage-collection pause of a few hundred milliseconds, or briefly saturated during a deploy, gets yanked from the pool. This is called **flapping**, and it's actively dangerous under load, because removing a backend concentrates its share of traffic onto the remaining, already-busy instances, which can push them past their own tipping point and cause them to fail their own health checks in turn - a cascading failure that can take an entire healthy fleet down from what started as one noisy probe. ## Thresholds too loose: slow detection Configuring checks too loosely - a long interval and a high failure threshold, such as checking every 60 seconds and requiring 10 consecutive failures - means a genuinely broken backend can sit in rotation minutes, continuing to serve errors or timeouts to real users, degrading the overall error rate and latency the whole fleet reports, before it is finally pulled. ## What production tuning looks like Production systems tune these deliberately asymmetric: it's common to - require several consecutive failures to mark unhealthy (avoiding flapping on transient blips) but only one or two successes to mark healthy again (so a recovered backend rejoins quickly); - pair active checks (for catching total outages fast, even with no traffic) with passive/outlier detection (for catching subtler, traffic-dependent breakage) rather than relying on either alone. AWS ALB target groups, Kubernetes readiness probes, and Envoy's outlier-detection filter are all concrete, widely used implementations of exactly this active-plus-passive combination.