When choosing a shard key strategy for a data store, what are the differences between lookup-based, range-based, and hash-based sharding, and what does each cost you?

## The three strategies A shard key strategy is the rule that maps a row's shard key value to a physical shard, and the three common strategies trade **locality**, **load balance**, and **operational control** against each other in different ways. - **Range-based sharding** assigns contiguous blocks of key values to the same shard — shard A might own customer IDs 1 through 1,000,000, shard B owns 1,000,001 through 2,000,000, and so on. Because rows with nearby key values live together, this strategy makes range-oriented operations cheap: "give me all orders between ID 500,000 and 500,100" or "give me events from the last hour" (when time is the key) can usually be satisfied by touching a single shard or a small contiguous set of them, without any cross-shard coordination. - **Hash-based sharding** instead applies a hash function to the key and uses the hash output (often mod the shard count, or via a hash ring) to pick a shard. Because a good hash function scatters similar or sequential inputs to unrelated outputs, writes for consecutive keys land on essentially random shards, which spreads load very evenly across the cluster. - **Lookup-based** (also called **directory-based**) **sharding** keeps an explicit mapping — a table or service that records "this key, or this key range, lives on this shard" — and every request consults that directory before routing. ## Why each one exists Each strategy exists to solve a different failure mode of the others. 1. **Range sharding** exists because ordered/range access patterns are extremely common (time-series data, paginated listings, ID-based backfills), and hash sharding makes those patterns expensive by scattering related rows across every shard. 2. **Hash sharding** exists specifically to defeat the load-skew problem that plagues naive range sharding: many real-world key spaces are not written to uniformly — IDs are usually assigned in increasing order, timestamps always increase, and both mean that under pure range sharding, essentially 100% of new writes target the single shard that currently owns the highest range, while older shards go idle. 3. **Lookup/directory sharding** exists for cases where neither pure math (range or hash) gives you the placement control you need — for example, deliberately isolating one enormous or noisy tenant onto its own dedicated shard, or supporting incremental, selective rebalancing without having to touch every key's mapping at once. ## What each one costs The trade-offs are concrete and show up immediately in system design. | Strategy | What you pay | What you gain | |---|---|---| | **Range sharding** | With range sharding, you pay in potential hotspots on monotonic keys | you gain cheap, single-shard range scans and simple, predictable shard boundaries that are easy to reason about and split | | **Hash sharding** | With hash sharding, you pay by losing locality — any range query, any "give me all rows for this related group" query, has to be sent to every shard and merged in the application layer (scatter-gather), which is slower and more complex than a local scan | but you gain near-uniform load distribution regardless of key access patterns, which is usually the bigger win for write-heavy systems | | **Lookup sharding** | With lookup sharding, you pay an extra network hop and a new critical dependency (the directory itself must be fast, consistent, and highly available, since every request needs it) | but you gain the ability to place data anywhere, override the default algorithm for exceptional keys, and rebalance incrementally by just updating map entries rather than rehashing the whole keyspace | ## Failure signatures in production In production, these strategies show up as distinct failure signatures. - A **range-sharded** system with a monotonic key shows one shard pegged at high CPU/IO while siblings idle, and the fix usually involves either changing the key (adding a randomized or reversed prefix) or manually splitting the hot range. - A **hash-sharded** system shows unexpectedly high latency and cost on range/list queries, because what should be a single-shard scan is silently fanning out cluster-wide, and the tail latency of these scatter-gather queries grows with shard count since it's bound by the slowest responder. - A **lookup-sharded** system's failure mode is usually availability or staleness of the directory itself — if the mapping service is down or serving a stale mapping after a rebalance, requests can be routed to the wrong shard or fail outright. ## Where it shows up A well-known real-world example: DynamoDB and Cassandra default to hash-based partitioning specifically to avoid the classic monotonic-key hotspot problem, at the documented cost that range queries across the partition key are not supported — you must model access patterns around the partition key up front or add a secondary index. Systems like Vitess, by contrast, support range-based (and directory) sharding for MySQL specifically because many workloads still need efficient range scans on the shard key, accepting the hotspot risk as something the schema designer must actively manage (e.g., by choosing a non-monotonic shard key) rather than something the sharding scheme solves automatically.