Walk through how Redis Sentinel detects that a master has failed and promotes a replica. What do the terms SDOWN and ODOWN mean in that flow?

## Step 0: what Sentinel is measuring Every Sentinel opens two connections to each monitored instance: a command connection (used for `PING`, `INFO`, `PUBLISH`) and a subscription connection to the `__sentinel__:hello` channel. `PING` goes out every second; `INFO` every ten seconds (more often during a failover). "Not replying" means: no `+PONG`, `-LOADING`, or `-MASTERDOWN` reply within the timeout — an error reply still counts as being alive but unhealthy for some cases, while `-BUSY` (a long-running Lua script) is treated as down after the timeout. ## Step 1: SDOWN — subjectively down If a Sentinel receives nothing valid for `down-after-milliseconds` (5–30 s typically; the tuning knob for how twitchy your cluster is), it flags the instance **SDOWN**. The word *subjective* is the point: this is one Sentinel's local belief. The Sentinel's own NIC, a one-sided partition, or a GC-frozen host can all produce SDOWN on a perfectly healthy master. A second SDOWN trigger exists for masters: a node that reports itself as a *replica* in `INFO` for long enough is also treated as down, because it is no longer serving the master role. SDOWN alone changes nothing about the topology. It emits a `+sdown` event, which is exactly what you want your alerting on. ## Step 2: ODOWN — objectively down The SDOWN-ing Sentinel now asks its peers `SENTINEL is-master-down-by-addr <ip> <port> <epoch> <runid>`. Each peer answers with its own view. When the number of Sentinels agreeing reaches the configured **quorum** (the last number on the `sentinel monitor` line), the master is promoted to **ODOWN** and `+odown` is published. Two important asymmetries: - **Only masters reach ODOWN.** Replicas and other Sentinels that are down stay at SDOWN, because nothing needs to be coordinated to stop using them. - **ODOWN is a fresh, non-sticky computation** based on replies received within a short window, so a flapping master does not accumulate stale votes. ## Step 3: leader election ODOWN authorizes a failover but does not perform it — if every Sentinel promoted a replica you would get several new masters. Sentinels therefore elect one leader using a **Raft-inspired algorithm with configuration epochs**. A Sentinel that sees ODOWN increments the epoch, votes for itself, and asks others for their vote (piggybacked on the same `is-master-down-by-addr` command, which returns the voted leader's run ID). Each Sentinel grants at most one vote per epoch, first come first served. The crucial rule: the winner needs **a strict majority of all known Sentinels** (`num_sentinels/2 + 1`), regardless of how small the quorum is. That is why quorum can never substitute for having enough live Sentinels — more on that in the sizing question. If no one wins the epoch (split vote), Sentinels back off a random delay and retry with a higher epoch. ## Step 4: choosing the replica The leader filters and ranks candidates: 1. **Discard** replicas that are SDOWN, disconnected, not seen recently, or whose link with the master has been down longer than roughly `down-after-milliseconds * 10 + (time master was down)`. 2. **`replica-priority`** (formerly `slave-priority`), lowest wins; **priority 0 makes a replica ineligible for promotion** — the standard way to pin a DR-site copy out of the running. 3. **Largest replication offset** — the replica that processed the most of the master's stream, minimizing data loss. 4. **Lexicographically smallest run ID** as a deterministic tiebreak. ## Step 5: promotion and reconfiguration The leader sends `REPLICAOF NO ONE` to the winner and polls its `INFO` until `role:master` appears. It then updates its own configuration with a new epoch, publishes `+switch-master <name> <old-ip> <old-port> <new-ip> <new-port>` — the event application clients subscribe to — and sends `REPLICAOF <new-master>` to the remaining replicas, at most **`parallel-syncs`** at a time so they are not all busy doing a full sync simultaneously and unavailable for reads. The other Sentinels learn the new configuration through hello messages and adopt it because it carries a higher epoch; each rewrites its own config file. When the old master returns, Sentinel makes it a replica of the new master — its unreplicated writes are discarded during resynchronization. `failover-timeout` bounds each phase; a stalled failover is aborted and retried, and it also rate-limits how soon the same master can be failed over again. ## Diagnosing it afterwards Sentinel's Pub/Sub log is the timeline: `+sdown` → `+odown` → `+try-failover` → `+vote-for-leader` → `+selected-slave` → `+promoted-slave` → `+failover-state-reconf-slaves` → `+switch-master` → `+failover-end`. If you see repeated `+sdown`/`-sdown` pairs with no failover, `down-after-milliseconds` is too aggressive for your network or your master is stalling (long-running commands, fork pauses, swap).