When does bucket_selector run, and why doesn't it make a terms aggregation cheaper?
answer
- It is HAVING in meaning only
- The script returns true or false
- Everything expensive already happened
- terms chose its buckets first
- Fewer than size buckets can come back
basics
~20 sbucket_selector runs during the reduce phase, after the parent aggregation has collected documents and already chosen its buckets. It only removes buckets from the response, so all the collection work and memory has already been spent, and it can never bring back a bucket the parent did not return.
solid answer
~50 s`bucket_selector` is a parent pipeline whose Painless script returns a boolean; buckets that evaluate false are removed from the response. It is the aggregation analogue of SQL's `HAVING` — but only in meaning, not in cost. It executes on the coordinating node during the reduce phase, **after** the shards have collected documents, built buckets and the coordinating node has merged them, so the memory and CPU were already paid. The practical consequence with a `terms` aggregation is a two-stage trap: `terms` first picks its top `size` buckets by its own ordering, and only then does `bucket_selector` prune among those. You get **at most** `size` buckets and often fewer, and a term that would have passed your predicate may never have been a candidate. To reduce cost you must filter documents in the query; to widen the candidate set you must raise `size`/`shard_size` and accept the memory, or use a threshold the bucket aggregation itself understands, such as `min_doc_count` on `terms`.
code
json · 17 lines{
"size": 0,
"aggs": {
"by_customer": {
"terms": { "field": "customer_id", "size": 10 },
"aggs": {
"spend": { "sum": { "field": "amount" } },
"big_spenders_only": {
"bucket_selector": {
"buckets_path": { "total": "spend" },
"script": "params.total > 10000"
}
}
}
}
}
}go deeper
Recall that bucket_selector keeps or drops whole buckets based on a script returning true or false, similar in spirit to SQL's HAVING.
Explain that it runs in the reduce phase over already-built buckets, so it changes the response but not the work, and that gap_policy governs buckets with missing values.
Diagnose the two-stage trap: terms truncates to size first, so the selector prunes an already-narrowed candidate set and qualifying terms can be missing entirely.
Own the tradeoff when reports need correct top-N by a computed metric: widen the candidate set and pay memory, push the restriction into the query, or enumerate buckets and filter outside the cluster.
## What bucket_selector is `bucket_selector` is a parent pipeline aggregation declared inside a multi-bucket aggregation. It takes a `buckets_path` map and a Painless `script` that must evaluate to a boolean. Buckets whose script returns false are dropped from the response: ```json "big_spenders_only": { "bucket_selector": { "buckets_path": { "total": "spend" }, "script": "params.total > 10000" } } ``` Semantically this is SQL's `HAVING`: a predicate over aggregated values rather than over documents. The comparison is a good one for explaining *what* it does and a bad one for reasoning about *what it costs*. ## When it runs A search fans out to every relevant shard. Each shard collects matching documents, builds its own version of the aggregation tree and sends partial results back. The coordinating node merges those partials in the **reduce phase**, and only then are pipeline aggregations evaluated. `bucket_selector` therefore sees a finished, merged bucket list and edits it. Everything expensive has already happened by that point: documents were collected on every shard, `terms` built its buckets using global ordinals, sub-aggregations ran for every bucket, and partial results crossed the network. Dropping a bucket at the end refunds none of it. A candidate who says "I used `bucket_selector` so the aggregation only processes the big customers" has the model backwards, and that is exactly the misconception the question probes. ## The terms truncation trap The sharper problem is correctness, not cost. A `terms` aggregation does not return every term; it returns the top `size` terms according to its `order` (document count descending by default), with each shard contributing its top `shard_size`. `bucket_selector` runs after that selection. So with `"size": 10` and a selector keeping buckets whose spend exceeds 10,000: - The candidate set is the 10 terms with the highest document counts. - Among those 10, any that fail the predicate are removed. - The response can contain anywhere from 0 to 10 buckets. - A customer with enormous spend across few orders may never have been in the top 10 by count, so the selector never saw them — and the report silently omits them. The fix depends on the shape of the predicate. If the threshold is on document count, `terms` has its own `min_doc_count` (and `shard_min_doc_count`), which the aggregation understands natively and applies as part of bucket selection rather than after it. If the threshold is on a computed metric, the only honest options are to enlarge the candidate set (`size` and `shard_size`), to order the `terms` aggregation by the metric that matters where that is possible, or to enumerate the full bucket space with a paging aggregation and filter client-side. Each of those trades memory or round trips for correctness, and saying so explicitly is the senior answer. ## What it is good at Used within its limits, `bucket_selector` is genuinely useful. It trims a bucket list you already intended to return — hiding buckets below a noise floor, keeping only accounts whose error rate crossed a threshold within an already-bounded set, or removing intervals whose value is null. It combines naturally with `bucket_script`: compute a ratio per bucket, then select on it. And `_bucket_count` in the path lets you filter parent buckets by how many sub-buckets they produced. It does **not** substitute for the query. If you can express the restriction over documents — a date range, a status, a customer tier — put it in the query or a `filter` clause, where it reduces the documents collected and therefore the actual work. ## Details worth knowing `gap_policy` applies here as elsewhere: with the default `skip`, a bucket whose referenced value is missing is treated as though it did not exist rather than being tested. The script must return a boolean (or something Painless treats as one); returning a number is a common error. And because the selector removes buckets after the parent computed them, other aggregations' summary values — for instance a sibling pipeline over the same bucket aggregation, or the `terms` aggregation's own `sum_other_doc_count` — are not recomputed to reflect the removal. Reading a total that includes buckets the response no longer shows surprises people; know which numbers are pre- and post-selection.
- A terms aggregation with size 10 and a bucket_selector returns only 3 buckets. Is that a bug?No — it is the documented behaviour. `terms` selects its top 10 buckets first, then the selector removes those failing the predicate, so the response holds at most 10 and frequently fewer. If you need ten *qualifying* buckets, raise `size` (and `shard_size`) so the candidate set is larger, and accept the extra memory and reduce cost that comes with it.
- How would you filter buckets by document count without a bucket_selector?Use the `terms` aggregation's own `min_doc_count`, with `shard_min_doc_count` if you also want shards to discard rare terms before sending results. These are understood by the bucket aggregation itself rather than applied afterwards, so they interact correctly with bucket selection instead of pruning a candidate set that was already truncated.
- Does bucket_selector reduce the memory an aggregation needs?No. Buckets are built on the shards and merged on the coordinating node before any pipeline aggregation runs, so peak memory is determined by the source aggregation's bucket count, not by how many buckets survive. The only levers that reduce that memory are a more selective query, a smaller `size`/`shard_size`, or a coarser grouping field.
It is a filter applied to a printed report, not to the query that produced it: the pages were already typeset, you are just crossing lines out.
saying these in an interview costs you the question
- Says bucket_selector makes the aggregation cheaper
- Expects exactly size buckets after selection
- Thinks it can surface terms outside the top size
- Uses it instead of a query filter on documents
- Writes a script returning a number rather than a boolean