As a sharded system grows and needs to add capacity, what does the process of resharding (changing the number of shards or moving data between them) actually involve, and what makes it risky to do on a live production system?

## What resharding is Resharding is the operation of changing how a sharded system's key space maps onto physical shards - adding shards to handle growth, retiring underused shards, or fixing an unbalanced distribution discovered after the fact - without taking the system offline. It sounds like a bookkeeping change, updating a mapping table or changing a modulus, but it is actually one of the **highest-risk operations** in a sharded system's lifecycle, because it means physically moving a large volume of live, actively-mutating data from one machine to another while the application keeps reading and writing it. ## The mechanical core, step by step The mechanical core of a safe online resharding process is a variant of the same pattern regardless of the underlying technology: **backfill plus dual-write or change-data-capture, plus cutover.** 1. **First,** you identify the subset of data that needs to move - a key range, a hash bucket, or a specific tenant's rows. 2. **Second,** you begin capturing every new write to that data as it happens, either by having the application dual-write to both the old and new location, or, the more robust and now more common approach, by tailing a change-data-capture stream off the source shard's write-ahead log or binlog (tools like Debezium, or built-in mechanisms in Vitess's VReplication) so every committed write is captured without relying on the application to remember to write twice. 3. **Third,** you run a backfill: a bulk copy of the existing historical data for that key range from the old shard to the new one, typically throttled so it doesn't starve production traffic of I/O capacity. 4. **Fourth,** once the backfill completes, the new shard is replaying live changes and catching up to the source in real time - you wait until the replication lag between old and new is effectively zero and stays there. 5. **Fifth,** the actual cutover: atomically flip the routing layer (the directory table, the consistent-hash ring assignment, or the proxy's shard map) so new queries for that key range go to the new shard instead of the old one. 6. **Sixth,** you keep the old shard's copy of the data around, read-only, for a rollback window in case something is wrong with the new shard, before finally decommissioning it. ## Why it stays risky The reason this is risky, even when every step above is followed carefully, comes down to a handful of concrete failure scenarios. - **The cutover moment itself is the sharpest edge.** Routing changes are rarely instantaneous across every client, proxy, and cache simultaneously, so there's a window where some requests are still being routed to the old shard while others have already moved to the new one - a write that lands on the old shard right after cutover can be silently lost if nothing is reading from there anymore, or a read immediately after cutover can miss a write that hadn't finished replicating from old to new. Systems mitigate this by making the cutover itself very short (flip a single routing key/version rather than reconfiguring every client individually) and by briefly pausing or queuing writes to the migrating range during the actual flip, accepting a tiny latency blip in exchange for correctness. - **A second risk is capacity.** During migration, both the old and new shard are live and serving/receiving traffic for the same logical data, and the source shard is additionally paying the cost of the backfill and CDC stream, so real spare I/O and CPU headroom is needed on the shard being migrated away from - attempting a migration on an already-saturated shard risks the backfill never catching up to incoming write volume, or degrading production latency for unrelated data still living on that same physical shard. - **A third risk is scale and duration.** Migrating a shard holding terabytes of data can take hours to days, during which the CDC/replication pipeline must remain reliable - any gap in the change stream, a dropped connection, or a restart that loses buffered events can silently desynchronize the target from the source in a way that isn't caught until data is queried and found missing or stale, which is why migrations are typically paired with a verification/checksum pass comparing row counts or hashes between source and target before cutover, not just trusting the pipeline. ## Why the earlier partitioning choice pays off here This is exactly why the choice of partitioning scheme made much earlier pays off or costs dearly at this stage: **consistent hashing** was specifically designed so adding or removing one shard only requires moving the data adjacent to that one change on the hash ring, rather than reshuffling the entire dataset the way naive `hash(key) mod N` does - this difference alone can be the difference between a resharding operation that takes hours and one that takes weeks. - Systems like **Vitess** automate much of this backfill-CDC-cutover choreography as a first-class 'reshard' workflow so operators don't hand-roll it per migration. - **DynamoDB and Bigtable** hide the mechanism entirely, splitting hot partitions automatically behind the scenes as part of normal operation rather than exposing resharding as a manual operator task at all.