What are the main production failure modes of service discovery systems - such as stale registry entries, a registry outage, or a 'thundering herd' on registry recovery - and how do teams mitigate them?

## Why these failure modes are inherent Service discovery infrastructure looks simple in a diagram - a registry, some instances registering, some clients looking things up - but in production it accumulates a specific, recurring set of failure modes that show up regardless of which tool (Consul, Eureka, Kubernetes) you use, because they're inherent to the problem of keeping distributed state about a constantly-changing fleet approximately correct. ## Stale data and detection lag The first and most fundamental is stale data / detection lag. Every failure-detection mechanism, whether active health checks or heartbeat leases, has a built-in window between an instance actually failing and the registry noticing and removing it - bounded by the check interval and failure threshold. During that window, the registry will keep handing out the dead instance's address to callers, who then experience connection refused, timeouts, or reset errors on what should have been a healthy call. This is unavoidable in principle - you cannot detect a failure faster than your detection mechanism polls or times out - so the correct mitigation isn't trying to shrink the window to zero (which just increases health-check load and false-positive risk from transient blips) but building the calling side to expect and handle it: - **client-side retries** against a different instance; - **short connect timeouts** so a dead instance doesn't hang the caller; - **circuit breakers** that stop routing to an instance after repeated failures even before the registry catches up. ## Registry unavailability The second is registry unavailability itself. A registry is itself a service, subject to crashes, overload, and network partitions, and if clients have no fallback when it's unreachable, every discovery lookup in the system fails at once - a single point of failure that can turn a localized problem (one registry node down) into a cluster-wide outage even though every actual service instance is perfectly healthy. Production-grade registries are built as replicated clusters specifically to survive this: - **Consul** runs a Raft consensus group among its server nodes, tolerating the loss of a minority without losing write availability. - **Eureka** peers replicate their registries to each other and, notably, favor availability over consistency during partitions - a Eureka server that's cut off from its peers will keep serving its own local (possibly stale) view rather than refusing to answer, a deliberate availability-over-consistency choice. On the client side, the standard mitigation is **caching**: a well-built discovery client keeps the last successfully fetched instance list around and keeps using it (with a warning/degraded-mode signal) if the registry becomes unreachable, so a registry blip degrades gracefully into working off slightly stale data instead of a hard outage. ## Thundering herd on recovery The third, often underappreciated, mode is the thundering herd on recovery. This shows up in a few concrete scenarios: - a registry restarts after maintenance or a crash and every one of thousands of instances tries to re-register simultaneously; - a network partition heals and every client that was operating off cached/degraded data tries to re-sync with the registry at once; - or a registry node rejoins a cluster and needs to catch up on state while also serving live traffic. The problem is that the exact moment the registry is most fragile (just starting up, still building its in-memory state, or mid-replication) is also the moment it gets hit with the largest concurrent load, which can cause it to fall back over, extending the outage. The standard mitigations mirror general thundering-herd fixes elsewhere in distributed systems: - **jittered exponential backoff** on reconnect/re-registration attempts so clients don't all retry in lockstep; - **staggering** health-check or registration intervals per instance rather than having them all fire on the same clock tick; - and sometimes explicit **rate-limiting** on the registry's registration endpoint. ## A real-world case where all three collide A concrete, well-known real-world instance of these failure modes colliding is the pattern of cascading discovery incidents teams have hit during AWS AZ-level network events on Eureka-based stacks: a partial network partition caused many healthy instances to miss heartbeats simultaneously, which without self-preservation mode would have caused Eureka to evict a large fraction of a healthy fleet, and then - once the partition healed - triggered a registration storm as everything tried to re-register at once. This is exactly the scenario Eureka's self-preservation mode and client-side caching were built to blunt: keep serving the last-known-good registry rather than emptying it out, and let clients ride out the blip on cached data rather than hammering the registry the instant it's reachable again. The general lesson for any registry choice is the same: assume the registry itself will fail, assume detection has a lag, and build the discovery-consuming side (retries, timeouts, circuit breakers, caching, backoff) to degrade gracefully rather than assuming the registry can be made perfectly fast and perfectly available.