A hash-sharded cluster running 8 shards needs to grow to 16 because storage per shard is filling up. What makes resharding — moving data to the new shard count — operationally risky, and how do teams typically do it with minimal downtime?

## What resharding is **Resharding** is the operation of changing a live sharded system's data-to-shard mapping — typically to add capacity — without taking the system offline. Mechanically, it starts with deciding the new topology (e.g., 8 shards becoming 16, or one overloaded shard being split into two), then moving the affected data from its old location to its new one while both the application and the routing layer stay aware of which rows currently live where. The naive approach — recomputing `shard = hash(key) % new_shard_count` for every row — is deceptively dangerous: because the modulus changed, the vast majority of keys resolve to a different shard number than before, even though most of them didn't need to move at all conceptually. A key that was on shard 3 out of 8 might now belong on shard 11 out of 16, with no relationship to its old placement. This means a naive resize forces something close to a full-cluster data migration rather than an incremental one, which is exactly why production systems avoid plain modulo hashing for anything that needs to grow, or why they lean on schemes (like consistent hashing, whose deeper mechanics belong to distributed-systems theory) or on shard-splitting instead. ## The staged migration The reason resharding exists as an operational discipline, rather than a one-line config change, is that a sharded system's whole purpose is to keep serving traffic continuously while its capacity grows — stopping the world to copy data would defeat much of the benefit of having sharded in the first place. So the standard approach decomposes the migration into stages that each individually preserve correctness. 1. **First, dual writes** (or an equivalent change-data-capture stream): once a target shard for a given key range is decided, every new write to that range is applied to both the old and new locations, ensuring the new location never falls further behind than "real time" from that point forward. 2. **Second, backfill**: a background job copies the pre-existing historical data for that range from old to new, typically throttled so it doesn't compete too aggressively with production traffic for IO and CPU. 3. **Third, verification**: before cutting reads over, the system (or an operator) checks that the new location's data actually matches the old location's — row counts, checksums, or spot-checks — because silently cutting over to an incomplete or diverged copy is worse than not migrating at all. 4. **Fourth, cutover**: reads for that key range are switched to the new location, usually gradually (a percentage of traffic, or one range at a time) rather than all at once, so problems surface on a small blast radius. 5. **Finally, cleanup**: once the cutover is confirmed stable, the old location's copy of that data is deleted and dual-writing stops. ## The trade-off The trade-offs of this approach are mostly about time and cost versus safety. It's slower and more operationally involved than a single atomic switch — a large resharding project can run for days or weeks — but it avoids downtime and gives multiple checkpoints where the team can pause or roll back if something looks wrong. Splitting an existing shard into two, when the sharding scheme supports it (e.g., range sharding where a shard just owns a sub-range of its old range going forward), is generally far cheaper than a full rehash, because only the data that's moving to the new shard has to move at all — the rest of the cluster's mapping is untouched. ## Failure modes The failure modes are concrete and well known operationally. - **Split-brain writes** happen if both the old and new locations accept independent writes for the same key without one being clearly authoritative — a client might read stale data from one location while another client's write only landed on the other. - **Read-your-write inconsistency** shows up during the cutover window: a user writes a row, and an immediate read is routed to whichever location hasn't yet received it, producing a confusing "my data disappeared" bug report. - **Prolonged dual-write windows** add sustained extra load and latency to every write in the migrating range, which can itself become a capacity problem if the migration runs too long. - **A router that isn't correctly aware** of exactly which key ranges have and haven't cut over yet is a common source of subtle, hard-to-reproduce bugs during the transition. ## Where it shows up Real systems build dedicated tooling for exactly this workflow rather than treating it as a one-off script. - **Vitess** (used for sharding MySQL at YouTube and Slack) has a formal "resharding" workflow built on VReplication that automates the dual-write/backfill/cutover sequence with built-in consistency checks. - **MongoDB's** sharded clusters run a background balancer that migrates data "chunks" between shards to keep load even, using a similar copy-then-cutover mechanism at the chunk level. - **Citus** (a sharding extension for PostgreSQL) provides shard rebalancing commands that move individual shards between nodes using logical replication to avoid downtime. The common thread across all of them is that resharding is treated as a first-class, tooled operation with explicit safety stages, precisely because doing it by hand with an ad hoc script is one of the highest-risk operations you can run against a production sharded database.