How do you page beyond 10,000 hits in Elasticsearch using search_after and a point in time?

## The problem search_after solves Offset paging (`from` + `size`) forces every shard to rebuild a top-`from + size` list for each page, so cost climbs with depth and Elasticsearch caps it at `index.max_result_window` (10,000 by default). `search_after` inverts the idea: instead of saying "skip 10,000 hits", you say "give me the hits that sort *after this exact position*". A shard can seek to that position and collect the next `size` documents, so page 1 and page 10,000 cost the same. ## Sorting and the tiebreaker `search_after` requires an explicit `sort`. The cursor is literally the `sort` array Elasticsearch returns on each hit, so the sort must impose a **total order**: if two documents share the sort value at a page boundary, the engine cannot tell which side of the boundary they belong on, and you get duplicated or silently skipped documents. The fix is a final tiebreaker field that is unique per document — an id with `doc_values`, a monotonically increasing sequence, or, in a PIT search, the built-in `_shard_doc` field. When a PIT is used and no tiebreaker is present, Elasticsearch appends `_shard_doc` automatically. ``` "sort": [ { "@timestamp": "desc" }, { "_shard_doc": "asc" } ] ``` ## Point in time: why it is the other half Without a PIT, `search_after` still works, but every page is executed against whatever the index looks like at that moment. New documents landing between page 3 and page 4 shift the sequence, so a user can see the same document twice or miss one entirely — the same drift that plagues offset paging over a live feed. A PIT freezes a view of the shards' segments. You open it, receive an id, and pass that id in the body of each subsequent search. Because the PIT identifies the target indices, the search request is sent to `/_search` with no index in the path. Each response returns a `pit_id` — possibly a refreshed one — and the well-behaved client always sends back the most recent id. When the walk finishes you `DELETE /_pit` with that id. `keep_alive` bounds how long the PIT survives between requests; it is extended by each search that uses it. Keeping a PIT open is not free: it pins the segments it references so they cannot be deleted after a merge, which costs disk space and file handles on the data nodes. Use short keep-alives that you extend as you go, rather than a one-hour window "to be safe", and always close explicitly instead of waiting for expiry. ## What the API will not let you do - **No `from`.** A `search_after` request must not carry a `from` parameter; the cursor *is* the position. - **No random access.** You can only step forward from a page you have already seen. There is no "jump to page 500", and going backwards means either keeping the previous cursors client-side or reversing the sort order and re-walking. - **No changing the sort mid-walk.** The cursor's meaning is defined by the sort; change the sort and the previous `sort` array is meaningless. The same applies to changing the query — the safe contract is that a cursor is only valid for the exact query, sort, and PIT that produced it. ## Slicing for parallel consumption When the goal is to consume everything as fast as possible rather than to serve a UI, a PIT search can be sliced: each worker issues the same query with a different `slice` id and walks its own `search_after` sequence. That gives parallel, non-overlapping coverage of one consistent snapshot — the modern replacement for sliced scroll. ## Designing the API around it Because `search_after` cursors are forward-only, an HTTP API built on them should expose an **opaque cursor token** ("next page" link) rather than a page number. Encoding the sort values, the PIT id, and a hash of the query into the token keeps clients from mixing a cursor with a different filter or sort — a mismatch that would otherwise return nonsense rather than an error. Decide up front how long a cursor stays valid, because it can outlive its PIT: a client resuming an hour later must be told to start over rather than served a stale or expired context. ## Rules of thumb Use `from`/`size` for the first handful of pages of a human-facing UI, `search_after` with a PIT for deep or unbounded paging and for infinite scroll, and sliced PIT walks for exports. If the interviewer asks what to do about the 10,000 limit and the answer is "raise the setting", that is the wrong half of the tradeoff.