Pipeline Aggregations

## Why a path language exists A pipeline aggregation consumes a value that some other aggregation produced. Since aggregations form a tree of named nodes, referencing that value needs an addressing scheme, and `buckets_path` is it. Getting it right is most of the practical difficulty of pipeline aggregations, and path errors are the most common failure people hit. ## The grammar **A bare name** refers to an aggregation reachable from the pipeline's own position. Inside a `date_histogram`'s `aggs` block, `"buckets_path": "sales"` means "the `sales` aggregation in this same block, evaluated for this bucket". **`>` descends** one level into a sub-aggregation. A sibling pipeline declared outside a bucket aggregation must use it to get in: `"sales_per_month>sales"` means "walk into the `sales_per_month` aggregation, and for each of its buckets take the `sales` metric". Paths can descend several levels: `"by_region>by_month>sales"`. **`.` selects a value** from a multi-value metric aggregation. A `stats` aggregation produces `count`, `min`, `max`, `avg` and `sum` in one node, so a pipeline aggregation must say which one it wants: `"sales_stats.sum"`. Single-value metrics such as `sum` or `avg` need no suffix. **Special keywords** address the bucket itself instead of a named aggregation. `_count` is the bucket's `doc_count` — invaluable, because it lets you compute per-document averages or thresholds without adding a redundant `value_count` aggregation. `_key` is the bucket key. `_bucket_count` gives the number of buckets in a named multi-bucket sub-aggregation, which is how you filter parent buckets by how many children they have — for example, keeping only sessions that visited more than three distinct pages. ## The map form Most pipeline aggregations consume exactly one value and take `buckets_path` as a string. The script-driven ones, `bucket_script` and `bucket_selector`, consume several and take a **map** from variable name to path: ```json "revenue_per_doc": { "bucket_script": { "buckets_path": { "total": "revenue", "docs": "_count" }, "script": "params.docs == 0 ? 0 : params.total / params.docs" } } ``` The keys of the map become the names under `params` inside the Painless script. This is worth stressing in an interview: the script does not name aggregations, it names the map keys, and mixing the two up produces a null-pointer failure at execution rather than a helpful path error. ## Relativity: the mistake everyone makes A path is resolved **relative to where the pipeline aggregation is declared**. Inside the bucket aggregation, `"sales"` is correct and `"sales_per_month>sales"` is wrong; outside it, exactly the reverse. Half of all "my pipeline aggregation does not work" problems are a path written for the other placement. The error message names the path that could not be resolved, which points straight at the fix once you know the rule. ## Missing values versus broken paths These two are different and are often confused. A **broken path** — naming an aggregation that does not exist, or descending into something that is not a bucket aggregation — is a request-level error; the search fails. A **missing value** — the path resolves fine, but a particular bucket has no value for it, as an `avg` yields on an empty bucket — is handled by `gap_policy`. The default `skip` treats that bucket as though it did not exist and continues from the next available value; `insert_zeros` substitutes zero, which is right for counts and sums but silently invents data for averages and rates. ## Practical notes Paths reference aggregation **names**, not field names; if the metric is named `sales` over field `price`, the path is `sales`. Because paths are resolved on already-computed results, a pipeline aggregation may reference **another pipeline aggregation's** output — a `bucket_sort` sorting on a `bucket_script` value is the everyday case. And on a bucket aggregation that truncates, such as `terms` with a `size`, the path resolves only over the buckets that came back; nothing in the path language reaches data the source aggregation did not return.