A globally distributed service uses DNS-based health checks, such as Route 53 health checks, to automatically fail traffic away from an unhealthy region. What makes this pattern riskier than an in-cluster readiness probe, and how would you design against flapping and split-brain?

## How the mechanism works, and what happens after the decision A DNS-based health check works by having the DNS provider, such as Route 53, poll a regional endpoint on a fixed interval and mark it healthy or unhealthy; a failover, weighted, or latency-based routing policy then removes the unhealthy region's records from the answers it serves to new lookups. The critical difference from an in-cluster mechanism is what happens after that decision is made. A DNS answer carries a TTL, but every resolver in the actual request path: - the client's OS stub resolver; - a local caching resolver; - an ISP's resolver; - a corporate proxy; - and sometimes an application that reuses a connection rather than re-resolving per request, can hold onto that answer well past its stated TTL, since many of these layers don't honor very short TTLs strictly or impose their own minimum caching duration. ## Why anyone uses it DNS-based failover exists as the traditional lowest-common-denominator mechanism for steering traffic across independent regions or even clouds, where there's no single load balancer spanning them the way a Kubernetes Service spans pods within one cluster. It's cheap, provider-agnostic, and universally supported, though lower-latency-propagation alternatives like global anycast load balancers exist for teams willing to take on that added infrastructure. ## The trade-off on two axes The trade-off against an in-cluster readiness probe is stark on two axes. | Axis | How the two compare | |---|---| | **Propagation latency** | An in-cluster readiness change takes effect in the Service's endpoint list in well under a second; DNS failover takes effect only as fast as caches actually expire across a heterogeneous, uncontrolled population of clients, which in practice is regularly minutes rather than seconds, regardless of the configured TTL. | | **Blast radius** | An in-cluster probe failure affects routing of traffic to one pod; a DNS failure or failover decision affects which of potentially millions of already-cached clients see which region, and because different clients' caches expire at different times, a genuinely split population can end up talking to two different regions concurrently — harmless for stateless, idempotent APIs, but dangerous for anything with regional session affinity, in-flight multi-step transactions, or data that's only eventually consistent across regions. | ## Flapping, and the asymmetric defense Flapping compounds this danger. If a health check oscillates near its failure threshold, because a region is intermittently slow rather than cleanly down, the DNS answer flips back and forth, but because of caching, clients don't cleanly follow the current state; each one sees whatever was cached when its resolver last asked, producing an incoherent client population split unevenly across regions and time. The standard defense is **asymmetric hysteresis**: - require a shorter sustained-failure window to trigger failover, since limiting damage from a real outage matters; - but require a much longer sustained-success window before allowing failback, since a premature failback into a region that's still shaky triggers another round of client splitting almost immediately. The health check driving failover should itself be simple and highly reliable, checked from multiple vantage points, because a false-positive failover is far more expensive and slower to undo than a false-positive pod restart. ## Failure modes that recur at this scale Several failure modes recur specifically at this scale. 1. **Low configured TTLs do not guarantee fast propagation**, because a meaningful fraction of real-world resolvers and client stacks disregard or exceed short TTLs, so observed failover completeness routinely lags the naive TTL-based expectation by minutes. 2. **A 'failback thundering herd'** can occur when many clients' cached answers expire around a similar time, or an automated health check flips the record back the instant its first successful poll comes in, sending a sudden surge of returning traffic into a region that just recovered and hasn't finished warming its own caches or connection pools, echoing the earlier startup-timing problem but at a much larger blast radius. 3. **Regional state splits.** And for any service with regional state, a shopping cart pinned to a region, a live session, a multi-step checkout, clients split across two regions mid-transaction can see inconsistent or lost state, which stateless APIs never have to worry about. ## A documented pattern A documented pattern used with Route 53 failover routing combines a fairly aggressive failover threshold, for instance three consecutive failed checks over ten-second intervals giving roughly thirty seconds to trigger failover, with a deliberately conservative failback approach implemented outside the automatic health check itself — often an on-call engineer manually re-enabling the primary region's record only after confirming full recovery, including cache warm-up and a sustained clean error-rate window, rather than letting the health check flip it back automatically the moment its first successful poll arrives. This asymmetry is precisely what avoids both the flapping and the failback-thundering-herd problems described above.