How do init_script, map_script, combine_script and reduce_script divide work in a scripted_metric aggregation?
answer
- four scripts, a fixed running order
- one runs per document, one per shard
- only one of them runs off-shard
- whatever crosses the network should be small
basics
~20 sinit_script runs once per shard to seed a state object, map_script runs once per matching document to accumulate into it, combine_script runs once per shard to produce the value sent over the wire, and reduce_script runs once on the coordinating node over the list of shard results.
solid answer
~50 s`scripted_metric` is a map-reduce escape hatch for metrics no built-in aggregation computes. Four Painless scripts run in a fixed order. `init_script` is optional and executes once per shard before collection, typically seeding the mutable `state` map. `map_script` runs for every matching document on that shard and accumulates into `state`, reading fields through `doc['field']`. `combine_script` runs once per shard after collection and returns the value that is serialized to the coordinating node — so it is where you shrink the per-shard state to something small. `reduce_script` runs once on the coordinating node with `states`, the list of every shard's combined result, and returns the aggregation's value. Because shard order is arbitrary, the reduce step must be order independent, and the state object is not covered by a circuit breaker, so an unbounded accumulator can exhaust heap.
code
json · 13 lines{
"size": 0,
"aggs": {
"weighted_score": {
"scripted_metric": {
"init_script": "state.num = 0.0; state.den = 0.0",
"map_script": "if (doc['weight'].size() == 0) return; double w = doc['weight'].value; state.num += doc['score'].value * w; state.den += w",
"combine_script": "return [state.num, state.den]",
"reduce_script": "double n = 0; double d = 0; for (s in states) { n += s[0]; d += s[1]; } return d == 0 ? null : n / d"
}
}
}
}go deeper
You are unlikely to be asked this. If it comes up, know that scripted_metric exists as a scripted escape hatch when no built-in metric aggregation fits.
Be able to name the four scripts and where each runs — per shard, per document, per shard again, then once on the coordinating node — and why the combine step exists at all.
Demonstrate that you know the hazards: unbounded state on the heap with no circuit breaker, serialization volume from combine, order independence in reduce, and non-additive metrics needing partial numerators and denominators.
Own the policy. Scripted aggregations are unreviewable performance risk at scale; decide when they are permitted, what alternatives (runtime fields, ingest-time precomputation, transforms) the platform provides, and how such scripts get reviewed and bounded.
## What it is for Every other metric aggregation computes a fixed thing. `scripted_metric` lets you compute an arbitrary one by supplying Painless scripts that follow a map-reduce shape. It is the last resort: slower than a native aggregation, harder to review, and unprotected by some of the safety machinery that native aggregations enjoy. Reach for it only after establishing that no composition of native aggregations and pipeline aggregations gets there. ## The four phases **init_script** — optional, runs once per shard before any document is collected. Its job is to prepare the mutable `state` object, a `Map` that persists for the duration of that shard's collection. A typical init is `state.totals = []` or `state.byKey = [:]`. **map_script** — required, runs once for every document matching the query on that shard. It reads document values, usually via doc-values access such as `doc['price'].value`, and folds them into `state`. This is the hot loop: whatever you write here executes millions of times on a large index, so allocation inside it is the main performance risk. **combine_script** — runs once per shard after collection finishes, and returns a value. That returned value — not `state` itself — is what gets serialized and sent to the coordinating node. Its purpose is reduction of volume: if `map_script` built a per-shard map of thousands of entries, `combine_script` is where you collapse it to the handful of numbers the reduce phase actually needs. Skipping that reduction and returning the whole state is the classic way to make a `scripted_metric` slow and memory hungry. **reduce_script** — runs once on the coordinating node. It receives `states`, a list holding each shard's combined result, and returns the final aggregation value that appears in the response. Because shards respond in nondeterministic order, the reduce logic must be commutative and associative in effect: any ordering of `states` must yield the same answer. ```json { "aggs": { "weighted": { "scripted_metric": { "init_script": "state.num = 0.0; state.den = 0.0", "map_script": "double w = doc['weight'].value; state.num += doc['score'].value * w; state.den += w", "combine_script": "return [state.num, state.den]", "reduce_script": "double n = 0, d = 0; for (s in states) { n += s[0]; d += s[1]; } return d == 0 ? null : n / d" } } } } ``` This computes a weighted average — a good illustration because the per-shard partial cannot be a single number. You must ship both the numerator and the denominator, since averages of averages are wrong. The same reasoning applies to any non-additive metric you implement this way. ## The hazards **Memory.** The `state` object lives on the shard's heap and is not accounted for by the request circuit breaker the way native aggregation data structures are. A `map_script` that accumulates one entry per distinct user id will happily grow until the node dies. Bound it deliberately, or reach for a native aggregation that has bounded structures. **Serialization cost.** Whatever `combine_script` returns crosses the network from every shard to the coordinating node. Returning large collections turns a search into a data transfer. **Doc access.** `doc['field']` reads doc_values and is fast; touching `params._source` inside `map_script` deserializes the stored source for every matching document and is dramatically slower. If you find yourself needing `_source` per document, the design is usually wrong. **Ordering.** Do not assume documents arrive in any particular order in `map_script`, or that `states` arrives in shard order in `reduce_script`. Logic that depends on ordering produces results that change between runs and between shard layouts. **Missing fields.** `doc['field'].value` on a document with no value throws. Guard with `doc['field'].size() != 0` or use `doc['field'].size() == 0 ? default : doc['field'].value`. ## Parameters and reuse All four scripts can read a shared `params` map declared on the aggregation, which is how you pass thresholds or weights without recompiling the script for every value. Painless compiles and caches scripts, so parameterizing rather than string-interpolating keeps the compilation cache effective — building a new script text per request can thrash it and hit the compilation rate limit. ## When not to use it Before writing one, check whether a `filters` aggregation with sub-aggregations, a `bucket_script` pipeline aggregation over existing metrics, a runtime field feeding an ordinary metric aggregation, or precomputing the value at ingest with an ingest pipeline solves the problem. Each of those is cheaper, reviewable, and protected by the normal safety limits. The strongest answer to this question ends by naming the alternatives you would exhaust first.
- Why must a scripted_metric that computes a weighted average return two numbers from combine_script?Because averages are not composable. A shard's local weighted average cannot be combined with another shard's without knowing how much weight each represents. Returning the numerator and denominator separately lets reduce_script sum both and divide once, which is the correct global answer. The same rule applies to any non-additive metric.
- What happens to memory used by the state object during map_script?It lives on the shard's JVM heap and is not tracked by the request circuit breaker that guards native aggregation structures. An accumulator that grows with distinct values — one entry per user id, say — can exhaust heap and destabilize the node before any limit intervenes, so bounding it is the script author's responsibility.
- When should you choose a runtime field or an ingest pipeline over scripted_metric?Whenever the value can be derived per document rather than across documents. A runtime field lets an ordinary metric aggregation do the work with normal safety limits; computing it at ingest is cheaper still because the cost is paid once instead of per query. Reserve scripted_metric for genuinely cross-document accumulation.
It is the map-reduce job you write by hand when the built-in reports do not cover your question: each shard tallies its own slice, hands up a small summary, and one node folds the summaries together.
saying these in an interview costs you the question
- Thinks reduce_script runs once per shard
- Returns the entire state map from combine_script
- Assumes documents reach map_script in a predictable order
- Reads params._source per document instead of doc values
- Believes the state object is protected by a circuit breaker