Why does Elasticsearch reject a search with from=10000 and size=10?

## What from and size do In the Elasticsearch search API, `size` is the number of hits to return and `from` is the number of leading hits to skip. `from: 20, size: 10` gives you the third page of a ten-per-page list. Both are pure result-set slicing — they do not change which documents match, only which slice of the ordered match list is returned to the client. `from` defaults to 0 and `size` defaults to 10. ## Why depth is expensive in a distributed index A search runs in two phases. In the **query phase** the coordinating node broadcasts the request to one copy of every shard. There is no global ordering available to any single shard, so each shard cannot know which of *its* documents fall into the caller's requested page — it must assume the worst and return its own best `from + size` results (document ids plus sort values) to the coordinating node. The coordinating node builds a priority queue of `number_of_shards × (from + size)` entries, sorts them into the true global order, discards everything outside the requested window, and then runs the **fetch phase** to load `_source` for the surviving `size` hits. So the work is proportional to `from`, not to `size`. `from: 9990, size: 10` on a five-shard index means 50,000 hit entries built, serialized, transferred, and merged so that ten documents can be shown. At `from: 1000000` the same request would be tens of millions of entries — enough to put real pressure on the coordinating node's heap and to trip circuit breakers. Nothing about this gets cheaper with more hardware: adding shards makes it *worse*, because the multiplier is the shard count. ## The cap: index.max_result_window To stop that failure mode, each index carries the dynamic setting `index.max_result_window`, default **10,000**, and Elasticsearch rejects any search where `from + size` exceeds it. The error message names the limit explicitly ("Result window is too large, from + size must be less than or equal to: [10000]") and suggests `search_after`. Note that it is the *sum* that is capped: `from: 9995, size: 10` fails just as `from: 10000, size: 10` does. A companion setting, `index.max_inner_result_window` (default 100), caps the same arithmetic for `inner_hits` and the `top_hits` aggregation, which is why you cannot page deeply inside nested results either. ## Why raising the setting is usually the wrong move `index.max_result_window` is dynamic, so it is tempting to `PUT` a bigger number and move on. That converts a clear 400-level error into an intermittent memory problem: the request now succeeds most of the time and takes down a coordinating node when several deep pages arrive at once, or when someone scripts a crawl of every page. The limit is a guard rail, not a bug. Raise it only for a bounded, known workload — for example an internal tool that occasionally needs page 1,200 of a small index — and raise it modestly. ## What to do instead - **`search_after`** is the supported deep-paging mechanism: sort by something with a unique tiebreaker, and pass the previous page's `sort` values to fetch the next page. Cost per page is constant regardless of depth, because each shard only has to find the first `size` documents *after* a given sort position. Pair it with a **point in time (PIT)** so pages stay consistent while the index is being written to. - **Scroll or a PIT-based export** when the goal is not paging at all but processing every matching document once, offline. - **Rethink the UI.** Almost no human clicks to page 900. Deep paging requests in production are usually a crawler, a broken "export" button, or an infinite-scroll client that never stops. Better facets, better relevance, or an explicit export endpoint removes the requirement instead of engineering around it. ## Common variations of the question Interviewers often follow up with "so what is the cost of `from: 10, size: 10000`?" — that is legal (the sum is 10,010, which fails, so make it 9,990) and cheap on the query phase but expensive on the *fetch* phase, because thousands of `_source` documents must be read and returned. Depth and page width are two different costs; the window bounds their sum.