On a consistent-hashing ring with only a handful of physical nodes, why can data and load still end up badly uneven, and how do virtual nodes (multiple ring positions per physical node) fix that?

## Why one position per node goes uneven Plain consistent hashing assigns each physical node a **single position** on the hash ring, and that node owns the arc stretching back to the previous node's position. The mechanism guarantees that key-to-node assignment is stable and that membership changes only move a bounded amount of data, but it says nothing about how big each node's arc is. Ring positions come from hashing an identifier through a hash function, and while a good hash function distributes outputs uniformly in expectation, with a small number of samples (a small number of nodes) the actual realized gaps between positions can vary a lot — the same statistical phenomenon as flipping a coin 5 times and getting 4 heads: individually fair, but small sample size means real variance from the 'expected' even split. With 8 physical nodes each getting one random ring position, it's entirely plausible for one node's arc to be **3-4x larger** than another's purely from the luck of hash placement, not from anything about the actual data distribution. That node then stores proportionally more data and, if load correlates with data volume, serves proportionally more traffic — a structural hot spot baked in by the placement scheme itself. ## How virtual nodes smooth the arcs Virtual nodes address this by breaking the **one-node-to-one-position mapping**. 1. Instead of hashing a node to a single ring position, the system hashes it (or a set of node-specific tokens) to many positions — common production numbers range from roughly 100 to 256 virtual nodes per physical node, though the exact count is a tunable trade-off. 2. Each virtual node behaves exactly like a regular consistent-hashing node: it owns the arc back to its ring-neighbor. 3. The only difference is that many of these virtual positions map back to the same underlying physical machine. Because each physical node now owns many small, independently-placed arcs instead of one large one, the total size it ends up with is the sum of many small random variables — and by the **law of large numbers**, that sum converges much more tightly around the 'fair share' (1/N of the ring) than a single random arc would. | Ring positions per physical node | Arc sizes | When a physical node joins | |---|---|---| | one, with 8 physical nodes | entirely plausible for one node's arc to be 3-4x larger than another's, purely from the luck of hash placement | one large contiguous arc from a single unlucky neighbor | | many | converges much more tightly around the 'fair share' (1/N of the ring) | small slices from many different peers simultaneously | ## The cost, and a second-order benefit The trade-off virtual nodes introduce is **bookkeeping and rebalancing granularity versus overhead**. - Every virtual node is a separate entry in the ring's metadata structure, so a cluster with 20 physical nodes and 200 virtual nodes each has 4,000 ring entries to track, propagate, and keep consistent across all participants. This is a real but generally modest cost — the metadata is small compared to actual data — and it's the price paid for smoother load distribution. - There's also a second-order benefit for rebalancing: when a physical node joins, instead of stealing one large contiguous arc from a single unlucky neighbor (which would mean that one neighbor does all the data-transfer work), the new node's many virtual positions are scattered across the whole ring, so it borrows small slices from many different peers simultaneously. This spreads the data-movement I/O across the cluster instead of concentrating it on one node pair, which matters operationally — a single-neighbor handoff can saturate that one node's network/disk I/O, while a many-peer handoff dilutes that cost. ## What undertuning looks like in production In production, undertuned virtual-node counts show up as persistent, unexplained imbalance in per-node disk usage or request rate that doesn't correlate with any actual hot key — dashboards where node utilization differs by 2-3x even though the workload's key distribution looks uniform — and the fix is increasing virtual-node count rather than anything at the application layer. - **Cassandra** is the canonical real-world example: it originally used one token per node, which produced exactly this kind of imbalance and made adding nodes to specific 'busy' spots on the ring operationally fiddly; it later moved to a default of many vnodes per physical node (the `num_tokens` setting, historically defaulting to 256, later tuned down for large clusters because too many vnodes has its own coordination overhead) specifically to get smoother distribution and multi-peer rebalancing behavior. - **Amazon's Dynamo paper** is the original source of the virtual-node technique for the same stated reason: uneven load from naive single-position consistent hashing on a heterogeneous, dynamically-changing set of nodes.