Your service runs in two federated Consul datacenters, and you want callers to use the remote datacenter only when no healthy instance exists locally. Which Consul discovery mechanisms would you weigh for that, and what does each one cost you?

## What federation gives you, and what it does not WAN federation joins the *server* tiers of two or more datacenters into a shared WAN gossip pool. Each datacenter keeps its own Raft peer set, its own leader, and its own catalog — nothing is replicated across the link. What federation buys is *forwarding*: a query naming another datacenter is routed to that datacenter's servers and answered there. The visible surface is a name: ``` web.service.dc1.consul # explicitly local web.service.dc2.consul # explicitly remote web.service.consul # this agent's own datacenter ``` Notice what is missing: there is no fallback. `web.service.consul` returns an empty answer when nothing local is healthy. Whoever wants failover has to implement it — either in the caller, or in the discovery layer. ## Option one: let the caller choose The caller resolves the local name, and on an empty answer resolves the remote one. This is explicit and completely observable — the client's own code and telemetry record which datacenter it chose. The costs are that every caller must implement the same logic identically, in every language you use, and the fallback decision is made per-caller with no coordination, so behaviour during a partial local outage is a matter of luck rather than policy. ## Option two: a prepared query with a failover policy A prepared query is a named, server-side query definition registered against the catalog. Its service block can restrict to healthy instances, filter by tag, sort by proximity, and — the part that matters here — carry a **failover** policy that names specific datacenters or asks for the nearest few. Callers resolve one name: ``` web-failover.query.consul ``` Consul evaluates it at query time: healthy local instances if there are any, otherwise instances from the first datacenter in the failover list that has some. Every caller gets identical behaviour by construction, the policy changes in one place, and nothing needs redeploying to change it. The trade-offs are the interesting part: **Opacity.** The name no longer tells anyone where the traffic went. A caller that suddenly has 90 ms of extra latency has no local signal saying "you are now talking to the other region". You have to build that observability deliberately. **Health becomes a global lever.** Failover fires on local health being empty. A misconfigured check, a dependency that every local instance shares, or an over-aggressive timeout can turn *all* local instances critical at once and send a datacenter's full traffic across the WAN — into a remote datacenter sized for its own load. That is a capacity event caused by a health-check bug. **Nothing prevents flapping.** Local health recovering and failing repeatedly moves traffic back and forth with no damping. Discovery-layer failover has no notion of hysteresis or of a human deciding to fail back. **WAN dependence.** The remote lookup itself crosses the link. A partition that takes out local instances *and* the WAN leaves you with nothing, and cross-datacenter data transfer usually has a bill attached. ## Option three: do not fail over at the discovery layer The alternative worth naming in a principal-level answer is keeping each datacenter's discovery strictly local and moving the failover decision up a tier — to whatever already steers traffic between regions before it reaches a datacenter at all. That decision is coarse-grained, observable, and typically reversible by a human, and it fails a whole region rather than one service at a time. Its cost is granularity: you cannot fail over a single misbehaving service without moving everything. ## How to decide Ask three questions: 1. **Is the remote instance actually usable?** Cross-datacenter latency, plus the data locality the service needs, decides this before anything else. Failing over a chatty service to a database it cannot reach quickly is worse than serving errors. 2. **How much do you trust local health?** Automatic failover is only as good as the signal that triggers it. If your checks are shallow or share a dependency, an automatic global lever is dangerous. 3. **Who should notice?** A prepared query hides the event; a caller-side or tier-above decision surfaces it. During an incident, an invisible failover is a debugging problem, not a save. A defensible position: use prepared queries where the remote datacenter is genuinely equivalent and the health signal is deep and per-instance; use an explicit tier-above decision where failing over is a big, rare, human-supervised event. What you should not do is add automatic cross-datacenter failover to the discovery layer simply because Consul supports it.