Explain how Consul keeps its service catalog and KV store consistent, and how it tracks cluster membership — i.e. Raft vs gossip.

**Two independent planes.** Consul deliberately separates *consensus* (agreeing on authoritative data) from *membership* (knowing which nodes are in the cluster and alive). They use different algorithms with different consistency/latency trade-offs. **Raft (consensus, servers only).** Raft is a leader-based consensus algorithm. Among the **server agents**, one is elected **leader**; the rest are **followers**. All writes — service (de)registrations that reach the catalog, KV puts, ACL changes — are routed to the leader, appended to a replicated log, and are **committed only once a majority (quorum) of servers have persisted them**. This yields **linearizable, strongly consistent** state: a successful write is immediately visible to consistent reads. Fault tolerance is quorum-based: a 3-server cluster survives 1 failure, 5 survives 2. If quorum is lost, the cluster can't commit writes (it favors consistency over availability for the catalog/KV). Followers can serve **stale** reads for lower latency if you opt in (consistency modes: `default`, `consistent`, `stale`). **Gossip (membership + failure detection, all agents).** Membership uses **Serf**, HashiCorp's implementation of the **SWIM** protocol (Scalable Weakly-consistent Infection-style process group Membership). Every agent — both clients and servers — participates. Nodes periodically pick random peers and exchange compact messages over **UDP (with TCP fallback)** to: - **detect failures**: a node probes a random peer; if no ack, it asks other nodes to probe indirectly before declaring it `failed` (reduces false positives from a single bad link); - **disseminate events**: joins, leaves, failures, and user events spread epidemically ('infection-style'), reaching the whole cluster in O(log N) rounds; - **maintain an eventually-consistent** view of membership — fast and highly scalable, but *not* linearizable. There are actually **two gossip pools**: a **LAN pool** (all agents within a datacenter, low-latency failure detection) and a **WAN pool** (only server agents across datacenters, for multi-DC federation). **How they interact.** Client agents hold no catalog/KV state; they use gossip for membership and **RPC-forward** discovery queries and KV operations to the servers, which serve them from the Raft-backed store. When gossip detects a node failed, associated health checks go critical and the catalog reflects it. So: *gossip is the fast, scalable liveness/membership signal; Raft is the slower, consistent source of truth.* **Why the split matters for Spring apps.** - **Catalog reads are consistent** (Raft), so discovery won't hand you an instance the leader never registered; but you can trade consistency for latency with stale reads. - **Failure detection is quick and decentralized** (gossip), so a crashed node is noticed cluster-wide without hammering the leader. - **Availability boundary**: if the server quorum is lost, *writes* (new registrations, KV changes) stall even though gossip still 'sees' nodes — an operational gotcha. **Edge cases / gotchas.** - **Split-brain protection**: Raft requires a majority, so a network partition leaves at most one side able to commit — no divergent catalogs. - **Cluster sizing**: even numbers of servers add no fault tolerance and waste replication; use odd counts (3/5). - **UDP blocked**: gossip degrades if UDP is firewalled (falls back to TCP, but tune ports). - **Anti-entropy**: each agent periodically reconciles its local services/checks with the catalog, self-healing drift. **When this knowledge is used.** Sizing a Consul cluster, reasoning about behavior during partitions, choosing read consistency modes for discovery, and explaining why a registration succeeded but a query on another DC lagged (WAN gossip + non-replicated catalogs per DC).