In PromQL, which queries are expensive to evaluate, and how do you decide what a recording rule should precompute?

## What a PromQL query costs The engine's work for one evaluation is close to **the number of series the selectors match, multiplied by the number of samples read from each**. A range query multiplies that again by the number of steps in the graph. Nothing else in the language moves the number as much as those two factors. Take a courtroom-scheduling platform on a 7-node cluster holding 2,340,000 active series, scraped every 15 seconds. - `rate(hearing_search_duration_seconds_bucket{job="search"}[5m])` matches 3,472 bucket series and reads about twenty samples from each: roughly 69,000 samples for one evaluation. Unremarkable. - `rate({__name__=~"hearing_.+"}[1h])` matches 384,000 series and reads about 240 samples from each: over 92,000,000 samples for one evaluation. A query engine carries a configurable ceiling on how many samples a single query may load, precisely so that this aborts instead of taking the server down with it. - Drawn as a six-hour graph, that same expression is not one evaluation but hundreds, one per step. Three shapes account for nearly all runaway queries. 1. **Wide selectors.** A regular-expression matcher on the metric name, or a selector with no `job` matcher at all, invites the engine to touch a large fraction of the store. Cardinality that is affordable at write time can be ruinous at read time. 2. **Long ranges.** Samples read per series scale with the bracketed duration divided by the scrape interval. Moving from `[5m]` to `[24h]` is a 288-fold increase in samples read per series, usually for a smoother line nobody asked for. 3. **Subqueries.** `max_over_time(sum by (courtroom) (rate(hearings_booked_total[5m]))[24h:1m])` evaluates its inner expression once per inner step — 1,440 times for that window — and each of those evaluations expands a five-minute range across every matching series. It is the one construct whose cost multiplier the reader has to work out for themselves. ## What a recording rule actually moves A recording rule evaluates one expression on a schedule and writes the result back as a new series. What changes is not the cost of the expression but **where and how often it is paid**. | | Ad-hoc expression | Recorded series | |---|---|---| | When it is evaluated | on every dashboard load and every alert evaluation | once per evaluation interval | | Series touched at query time | thousands to hundreds of thousands | one per output label combination | | Cost of ten people opening the dashboard | paid ten times | no extra cost | | History available | as far back as the raw data | only from when the rule was created | The second row carries most of the value. `sum by (courtroom) (rate(hearings_booked_total[5m]))` expands 3,472 series and collapses them to 41; recorded, the dashboard reads 41 series and a handful of samples each. The read path stops being proportional to the size of the fleet. What it does not do is make anything free. The rule still runs on the same server at every evaluation interval, forever. If the expression is heavy enough the rule falls behind its schedule, and a rule that falls behind delays everything that reads it. Recording an expensive query converts an unpredictable cost per viewer into a predictable cost paid once. It does not remove the cost. ## Deciding what to precompute For a platform whose on-call rotation is three people, the discipline that survives contact with reality is narrow. 1. **Anything an alert evaluates.** Alert expressions run on a fixed schedule forever, so a slow one is a reliability problem rather than a performance annoyance. 2. **Anything the incident dashboards read.** Those are opened repeatedly, under time pressure, by the same three people, often simultaneously. That multiplier is exactly what a recorded series removes. 3. **Expressions that collapse enormous fan-out into a small, stable label set.** The saving is the ratio between input series and output series, so recording something that stays at full cardinality saves almost nothing and doubles the storage. And the things not to record: - Exploratory queries. If nobody has run it twice, it does not need a rule, and you will accumulate rules nobody dares delete because nobody knows what reads them. - Anything already cheap. Every rule has a permanent cost in evaluation time, in storage, and in one more name people must learn. - The raw detail itself. A rule produces an aggregate; keeping the underlying series is a retention decision, not a query one. Two organisational points outweigh the query mechanics at this level. First, a recorded series is a **published interface**: other teams build dashboards and alerts on it, so its name and its label set deserve the same care and the same deprecation manners as an API, which is the whole reason for the convention of naming a recorded series after the aggregation level, the source metric and the operations applied. Second, every recorded series is a second copy of a truth that also exists in raw form, and when the two disagree — because the expression changed, or because the rule was created after the incident someone is now trying to explain — people trust the wrong one. Keep the set small enough that the team can hold all of it in their heads.