How do you choose the primary shard count for a new Elasticsearch index whose future size you don't know?

## Why this is a judgment question `index.number_of_shards` is final. Everything about the decision follows from that: you are picking a number that will govern an index for its entire life, usually before you have seen production data or production query patterns. There is no formula that produces the right answer, so the real skill is designing so that being wrong is cheap. ## Step one: measure, do not guess The only number that matters is how much data a single shard of *your* documents, with *your* mappings, can hold while still meeting *your* latency objective. Get it like this: 1. Create a one-primary, zero-replica index with the production mapping and analysis chain. 2. Load a representative sample — same field cardinality, same document size, same proportion of nested documents or vectors, because all of those change the answer dramatically. 3. Replay a realistic query mix, including the expensive aggregations and the deepest paginations, at realistic concurrency. 4. Grow the shard and watch p99 latency, heap and merge activity. The size where latency crosses your objective is your shard-size budget. A log workload with keyword filters and date ranges will tolerate a very different shard size from a product catalogue with heavy aggregations and highlighting, or an index carrying dense vectors. This is why generic guidance is a broad band in the tens of gigabytes rather than a single number: it is the range most workloads land in, not a substitute for measuring yours. ## Step two: project the volume and divide Estimate data volume at the retention horizon you actually keep — daily ingest multiplied by retention, plus the index overhead over raw document size, which you now know from the benchmark. Divide by your measured shard budget, then round up for growth headroom. Remember to count replicas when checking disk and node capacity: each replica is another full copy. ## Step three: apply the structural constraints Two hard bounds frame the number: - **Lower bound: node spread.** An index cannot occupy more data nodes than it has shard copies. If you intend to spread an index across eight data nodes, one primary with one replica cannot do it. Shard count is also what allows a single query's work to be spread across machines. - **Upper bound: overhead.** Every shard copy costs heap, file handles, merge threads, cluster-state entries, and one task per search. Guidance to stay under roughly 20 shards per gigabyte of JVM heap on a node, and the `cluster.max_shards_per_node` default of 1000 open shards per data node, are the guard rails. An inflated count buys nothing and degrades every query. Between those bounds, prefer the smaller number that still meets the spread requirement. Over-sharding "just in case" is the more common and more expensive mistake, because its cost is paid on every single search forever, whereas under-sharding is fixable with a split or a reindex. ## Step four: design so the bet is not permanent This is what separates a principal-level answer: - **Append-only or time-series data** (logs, metrics, events, orders) should not live in one growing index at all. Roll over to a new index on a primary-shard-size condition, keep the shard count in a shared template, and let retention delete the old ones. The shard count is then revisited implicitly on every roll, and a wrong choice ages out instead of becoming permanent. - **Mutable corpora** (a product catalogue, a user directory) do live in one index, so sizing matters more — but they are also usually small enough that one or a few shards is right, and a reindex behind an alias is a routine, rehearsed operation. Always front such an index with an alias from day one, so the swap costs nothing when the day comes. - **Set `index.number_of_routing_shards` deliberately** if you expect an index to need splitting later; it is final too, and it caps the split factors available to you. ## Common bad heuristics to challenge - *"One shard per data node."* It guarantees spread but ignores data volume entirely, and it turns node additions into a shard-count problem. - *"Pick the maximum you might ever need — extra shards are free."* They are not; the fixed per-shard overhead is paid whether the shard holds 200 MB or 50 GB. - *"We will just split it later."* Split requires a write block and a compatible routing-shard configuration; it is a planned maintenance operation, not a live resize. - *"Match the shard count to CPU cores."* Search parallelism is bounded by shard count, but so is nothing else about capacity; the data volume is the primary driver. ## How to present the decision Write it down: the measured shard budget, the volume projection and its assumptions, the resulting count, the node-spread requirement it satisfies, and the escape hatch you have preserved (alias plus reindex, rollover, or a routing-shard configuration that allows splitting). A shard count with that reasoning attached is defensible; a number with no reasoning is the thing that gets copied into the next team's template and causes the next oversharded cluster.