You need to add a fourth master to a live three-master Redis Cluster and give it a fair share of the keyspace. Describe the operational procedure with redis-cli's cluster tooling and how you verify the result.

## What scaling out actually means here Redis Cluster does not rebalance itself. Slots are assigned explicitly and stay put until someone moves them, so "add a shard" is two independent steps: make the node a cluster member, then hand it slots. Keeping those steps separate is a feature — a joined-but-empty master serves no traffic, so you can add it well ahead of the migration window and roll back by simply never resharding to it. ## Step 1 — join ``` redis-cli --cluster add-node 10.0.0.9:6379 10.0.0.1:6379 ``` The first address is the newcomer, the second is any existing cluster member used as the entry point. The tool checks the new instance is empty and has `cluster-enabled yes`, then issues `CLUSTER MEET`. Gossip propagates the membership. `CLUSTER NODES` will now list it as a master with an empty slot range. ## Step 2 — move slots Two interfaces to the same primitive loop: **Targeted reshard**, when you know exactly what you want: ``` redis-cli --cluster reshard 10.0.0.1:6379 \ --cluster-from <id-a>,<id-b>,<id-c> \ --cluster-to <id-new> \ --cluster-slots 4096 \ --cluster-yes ``` With four masters an even split is 4096 slots each, drawn proportionally from the three incumbents. Run interactively (omit the flags) and the tool prompts for the same values, printing the slot plan before executing. **Rebalance**, when you want the tool to compute the plan: ``` redis-cli --cluster rebalance 10.0.0.1:6379 --cluster-use-empty-masters ``` `--cluster-use-empty-masters` is essential — without it the new zero-slot master is ignored, which is the single most common reason "rebalance did nothing". Useful modifiers: `--cluster-weight <id>=<w>` for heterogeneous nodes, `--cluster-threshold <pct>` for how far off balance is tolerable, `--cluster-pipeline <n>` for the number of keys per MIGRATE batch (default 10; raising it speeds the copy but lengthens each blocking call), and `--cluster-timeout` for the MIGRATE timeout. For each slot the tool performs the manual protocol: `CLUSTER SETSLOT <slot> IMPORTING` on the target, `MIGRATING` on the source, then `CLUSTER GETKEYSINSLOT` + `MIGRATE ... KEYS ...` until the slot is empty, then `CLUSTER SETSLOT <slot> NODE <target>` on the masters. Traffic keeps flowing; clients see ASK redirects for keys already moved and MOVED once ownership commits. ## Step 3 — replica for the new master ``` redis-cli --cluster add-node 10.0.0.10:6379 10.0.0.9:6379 \ --cluster-slave --cluster-master-id <id-new> ``` Until this runs, a quarter of your keyspace has no failover target. If `cluster-require-full-coverage` is `yes` (the default), losing that master takes the **whole cluster** down for writes, not just its slots — so the replica is not optional in production. ## Step 4 — verify - `redis-cli --cluster check 10.0.0.1:6379` — the important line is `[OK] All 16384 slots covered` and the absence of `[WARNING] Node ... has slots in migrating state`. Open slots mean an aborted migration; `--cluster fix` resolves them. - `redis-cli --cluster info` — keys and slot counts per node; confirms the new node actually received data. - `CLUSTER INFO` on any node — `cluster_state:ok`, `cluster_slots_assigned:16384`, `cluster_known_nodes` matching your inventory. - Application side: watch error rates for `MOVED`/`TRYAGAIN`, and confirm clients converged (a client that never refreshes topology will keep paying a double round trip; see its metrics or latency). ## Pacing and risk Resharding is a live-traffic operation and its cost is dominated by `MIGRATE`, which blocks both the source and target event loops per batch. Practical discipline: move slots in chunks rather than 4096 at once; keep `--cluster-pipeline` modest if you have large values; run during a trough; watch `SLOWLOG` and p99 latency between chunks; and know that a very large single key (a multi-gigabyte hash) can stall two masters for seconds, so find and split those before you start. Also ensure `cluster-node-timeout` is comfortably larger than your worst MIGRATE stall, otherwise a blocked master can be mistaken for a failed one and trigger a failover in the middle of your reshard. ## Scaling in The mirror image: reshard the departing node's slots away, then `redis-cli --cluster del-node <any-node> <id>`. `del-node` refuses to remove a master that still owns slots, which is a useful guard rail.