In an active-active geode deployment - multiple regions, each running a full copy of the backend and application data, any region able to serve any request - how does a client's request typically get routed to a nearby geode, and what happens automatically when one geode's region suffers an outage?

## The layer in front of the fleet Getting a request to the right geode, and getting it away from a broken one, is the job of a **routing/health-check layer** that sits logically in front of the fleet of regional deployments. There are a few concrete mechanisms teams use for the routing half. - **Latency-based or geographic DNS** steers a client to a region based on either the client's resolved location or measured network latency from various vantage points, resolving the same hostname to different regional IP addresses depending on who's asking. - **Anycast** takes a different approach: the same IP address is announced from multiple regions, and ordinary internet routing (BGP) delivers each packet to whichever announcing location is topologically nearest, so 'nearest geode' falls out of the network layer itself rather than DNS. - **Many teams instead use a managed global traffic-manager or global load-balancer product** that offers a hybrid of geo-proximity, latency, and weighted routing policies, plus built-in health checking, so they don't have to hand-roll DNS logic. Whichever mechanism is chosen, the routing decision usually isn't purely static - it's continuously informed by health signals, which is the second half of the picture. ## Health checks come in layers Health checking in this context is layered. - **A basic check** might just confirm the region's load balancer or network path is reachable. - **A more meaningful check** calls an application-level health endpoint that verifies the service process is up and can reach its own dependencies (database connection, cache, downstream services). - **The most thorough setups run synthetic transactions** - a scripted request that exercises a real code path - on a schedule, so that a geode which is technically 'up' but returning errors or badly elevated latency for real traffic still gets flagged. These checks run continuously and independently of any actual outage, which is what makes automatic failover possible: the system already knows a geode is unhealthy within the health-check interval (commonly tens of seconds), not after users start complaining. ## What an evacuation actually looks like When a geode's checks start failing, the routing layer marks it out of rotation and stops sending new requests there. Existing in-flight requests to that geode still fail or time out - a real user-facing blip, not a magic instant fix - but new requests get routed to the remaining healthy geodes, chosen the same way, by proximity, latency, or configured weight. Because every geode was already running a full copy of the application and already held a replicated copy of the data, this reroute is a matter of **DNS TTL expiry** or **traffic-manager reconvergence**, seconds to low minutes depending on the mechanism and configured TTLs, not a cold start of new infrastructure. That's the central operational payoff of active-active over active-passive: there's no 'promote the standby, warm its caches, redirect traffic' runbook to execute under pressure, because the standby doesn't exist - every region was already a full peer. ## The edges of the automation The trade-off is that this automation has real edges. - **DNS-based routing is bounded by how aggressively clients and resolvers honor TTLs** - a very low TTL improves failover speed but increases DNS query volume and cost, while a high TTL means some clients keep hammering a dead region for longer than the health check alone would suggest. - **Anycast reconvergence depends on how quickly BGP routes propagate**, which is mostly out of the application team's control. - **Health checks themselves can be wrong in both directions**: too shallow a check (just 'is the load balancer up') can leave a genuinely broken geode in rotation because the check doesn't exercise the failing code path, while too aggressive a check (flagging on transient latency blips) can cause 'flapping,' where a geode gets pulled in and out of rotation repeatedly, adding instability instead of removing it. Teams typically tune this with **hysteresis** - requiring several consecutive failed checks before pulling a geode out, and several consecutive successes before adding it back - to avoid flapping. ## Failure modes Failure modes commonly seen in production: - a **'silent brownout,'** where a geode is slow or erroring for a subset of endpoints but the shallow health check still passes, so traffic keeps flowing there and users in that region see a degraded experience with no automatic mitigation; - a **health-check dependency loop**, where the health endpoint itself calls a downstream service that's degraded, causing the check to fail (and traffic to reroute) even though the core service is actually fine, amplifying a small dependency problem into a full regional evacuation; - and **DNS-caching issues** at the client or corporate-proxy level that keep sending some fraction of users to an already-evacuated region well past the intended TTL. ## Where it shows up A concrete real-world pattern: large content and API platforms commonly pair a global traffic-manager or anycast-fronted CDN/API gateway with regional backend geodes, configuring latency-based routing plus multi-level health checks (network, app, and synthetic) with a short but not razor-thin TTL and a few-consecutive-failure threshold before evacuating a region - trading a little failover speed for protection against flapping on transient blips.