Redis Sentinel

## Why discovery is the client's job Sentinel is not in the data path. It cannot silently move traffic; all it can do is *tell* whoever asks where the master currently lives, and *announce* when that changes. So high availability with Sentinel is only as good as the client integration. A perfectly healthy Sentinel quorum plus a hardcoded IP in `application.yml` equals an outage that Sentinel dutifully logs as successfully resolved. ## The two APIs the client uses **Request/response:** ``` SENTINEL get-master-addr-by-name mymaster 1) "10.0.0.11" 2) "6379" ``` Any Sentinel can answer. Because a Sentinel that is itself partitioned may return a stale address, robust clients treat the answer as a hint and confirm with `INFO replication` that `role:master` is present before using it for writes; if not, they try another Sentinel. **Push:** ``` SUBSCRIBE +switch-master # message: mymaster 10.0.0.10 6379 10.0.0.11 6379 ``` The payload is `<master-name> <old-ip> <old-port> <new-ip> <new-port>`. On receipt the client must **close its existing connections** to the old address, not merely note the new one — an established TCP connection to a demoted node stays usable and will happily accept reads while rejecting writes. Related events worth reacting to: `+sdown`/`-sdown` and `+slave` for read-replica pools, and `+redirect-to-master`. ## What the client library actually does for you Given a Sentinel configuration, a good client: - Maintains a pool of Sentinel connections and rotates through them; a Sentinel being down must not block startup. - Reorders its Sentinel list so the last responsive one is tried first. - Keeps one subscriber connection for failover events. - Refreshes the topology periodically as a backstop in case a `+switch-master` message was missed while disconnected. - Exposes separate "master" and "replica" connection intents so reads can be routed to replicas while writes always go to the resolved master. Spring Data Redis, for example, takes a `RedisSentinelConfiguration` with the master name and Sentinel nodes; Lettuce underneath does the resolution and event subscription. ## The failure modes of hardcoding 1. **During the outage** — connections to the dead master hang until the socket timeout; without a sane `connectTimeout`/command timeout, threads pile up and the failure spreads through your service. 2. **After the failover** — the old master returns as a replica. Reads succeed (which is why this bug survives smoke tests), writes fail with `-READONLY`. Some apps interpret that as a transient error and retry forever. 3. **Split-brain window** — clients still attached to a partitioned old master write data that is discarded when it resynchronizes. Sentinel-aware clients close those connections as soon as they see `+switch-master`; a hardcoded client does not. 4. **Silent staleness** — if you point reads at a fixed replica IP that gets promoted, you have now made your "read replica" the master and are unknowingly sending read load to it. ## Alternatives when the client cannot speak Sentinel Some stacks (a legacy driver, a sidecar, a tool that only accepts one host) cannot do Sentinel discovery. Two standard workarounds: - **`client-reconfig-script`** — Sentinel invokes a script on the leader Sentinel during failover with the master name, role, state, and old/new addresses. The script moves a floating VIP, updates a DNS record with a short TTL, or reconfigures a proxy such as HAProxy. Caveats: it runs only on the Sentinel that led the failover, it must be idempotent, and DNS TTLs plus JVM DNS caching add real delay. - **A proxy layer** (HAProxy, Envoy, or a Redis-aware proxy) that health-checks `role:master` and exposes a single stable endpoint. This restores a single point of failure unless the proxy is itself made HA, and adds a hop of latency. ## Testing it The only trustworthy verification is a **forced failover under load**: run `SENTINEL failover mymaster` while your application is writing, and assert that write errors stop within your target window and that no thread pool is exhausted. Many teams discover their timeouts and retry policy — not Sentinel — are the reason a 10-second failover becomes a 3-minute outage.