When do you need an Elasticsearch composite aggregation instead of a plain terms aggregation?

## The gap composite fills A `terms` aggregation has no pagination. `size` is a top-N cutoff, not a window: there is no `from`, no cursor, no way to say "the next 100 terms after those". That is a deliberate consequence of how it works — each shard nominates only its own top candidates, so the notion of "page 2" is not even well defined. If your requirement is exhaustive enumeration of groups, `terms` is structurally the wrong tool no matter how high you push `size`, and pushing `size` high enough to cover a high-cardinality field is how you trip a circuit breaker. The `composite` aggregation exists for exactly this case. It produces buckets in a defined, stable order and hands you a cursor to continue from. ## Sources and the composite key `composite` takes an ordered array of `sources`. Each source names an output key and wraps a value source: `terms` (a field's values), `histogram` (fixed numeric buckets), `date_histogram` (time buckets) or `geotile_grid`. The aggregation emits one bucket per distinct *combination* of the source values, and each bucket's `key` is an object with one entry per source: ``` { "key": { "user": "u-914", "day": 1690848000000 }, "doc_count": 12 } ``` This is a multi-column `GROUP BY user_id, day` — and note that it is genuinely different from nesting a `terms` inside a `terms`, which produces a tree with per-level truncation at every node. Composite produces a flat, complete cross-product of the combinations that actually occur. Each source has its own `order` (`asc` or `desc`), and the buckets come back sorted by the composite key using those per-source directions. `missing_bucket: true` on a source makes documents lacking that field appear in a bucket with `null` for it rather than being dropped — without it, a document missing any one source value is excluded from the aggregation entirely, which is a common silent data-loss bug in export jobs. ## Paging with after_key `size` on a composite aggregation is the page size. The response includes `after_key`, the composite key of the last bucket on the page. Send the next request identically but with `"after": <that key>` inside the composite object, and you receive the buckets that sort strictly after it. Repeat until a response comes back with no buckets — that empty page, not an `after_key` being absent, is the reliable termination condition. Because the ordering is by composite key rather than by an internal cursor, the mechanism is the aggregation analogue of `search_after`: it is stateless on the server, so there is no scroll context to expire and no server-side memory held between pages. ## The two costs **No ordering by count.** Composite can only sort by its source values. You cannot ask it for "the 10 categories with the most documents" — it has no idea which those are until it has emitted them all. If you want top-N by count, use `terms` (and accept its approximation), or page everything out of composite and sort client-side, which only makes sense if the total bucket count is modest. **No snapshot across pages.** Each page is an independent search. On an index that is being written to, documents indexed between page 3 and page 4 can add buckets you have already passed, or change counts you have already recorded. For an export that must be internally consistent, open a point-in-time and issue every page against it, so all pages see the same view of the index. Sub-aggregations inside a composite are allowed and useful (a `sum` per group, for instance), but remember they are computed per page against whatever the index looked like at that moment. There is also a plain throughput cost: `composite` streams every bucket, so an export over a very high-cardinality field is many round trips. Choose the page `size` deliberately — large enough that you are not paying request overhead per handful of buckets, small enough that a page's buckets and sub-aggregation state fit comfortably. ## When you should not reach for it If the answer is a dashboard's top 10, use `terms`. If you need the same exhaustive rollup repeatedly and cheaply, materialise it once — a transform or a scheduled job that writes per-key documents into their own index turns a multi-page scan into a single lookup, and gives you exact counts as a side effect. Composite is the right tool for a one-directional walk of the whole key space; it is a poor substitute for a well-chosen top-N or for pre-aggregation.