Why would you add a Prometheus recording rule, and what does its rule group's interval control?

A recording rule is the other half of Prometheus's rule engine. It shares the file format and the evaluation loop with alerting rules, but instead of producing an alert it produces a **new time series** that is written back into the store. ## The shape of a recording rule A recording rule has exactly two required fields, plus optional labels: ```yaml groups: - name: membership-recording interval: 30s rules: - record: job:gym_checkins:rate5m expr: sum by (job) (rate(gym_checkins_total[5m])) ``` At every evaluation Prometheus runs `expr`, takes each sample in the result, renames it to the value of `record`, keeps the result's labels (plus any `labels` you add on the rule), and appends it at the evaluation timestamp. From then on `job:gym_checkins:rate5m` is an ordinary series that any query, dashboard or alerting rule can select. ## Why precompute at all Three reasons, in descending order of how often they actually decide it: 1. **Query cost.** An aggregation that fans across a 2.3-million-series store is expensive every single time it runs. On a membership platform where one team's services generate roughly seventy per cent of the series, their team dashboard alone can re-run the same heavy aggregation from a dozen panels on every refresh. A recording rule collapses all of that into one evaluation per interval. 2. **Long ranges.** A query over thirty days against raw series has to read every sample in the range. Against a pre-aggregated series recorded every 30 seconds it reads orders of magnitude fewer points, which is often the difference between a dashboard that loads and one that times out. 3. **One definition, many consumers.** When the dashboard panel, the alerting rule and the weekly report each re-type the same expression, they drift. Recording the expression once makes the alert and the graph provably the same number. ## Groups: interval and ordering Every rule sits inside a group, and the group — not the individual rule — is the unit of scheduling: | Setting | Scope | What it does | |---|---|---| | `evaluation_interval` (global) | whole server | Default cadence for every group that does not override it | | `interval` (per group) | one group | Overrides the global cadence for all rules in that group | | file order within a group | one group | Rules evaluate **sequentially**, top to bottom | Two consequences follow, and interviewers ask about both. **Ordering is a real guarantee inside a group and a real hazard across groups.** Because rules in a group run one after another against the same evaluation, a rule can reference a series produced by an earlier rule in the *same* group and get this cycle's value. Put those two rules in different groups and you lose that: groups are scheduled independently, so the dependent rule reads whatever the previous cycle happened to leave behind, or nothing at all when the store is fresh. Chained recording rules therefore belong in one group, in dependency order. **A group that cannot finish within its interval falls behind.** Sequential evaluation means the group's total runtime is the sum of its rules. If that exceeds `interval`, evaluations start late and eventually get skipped, and your recorded series develops gaps that look like an outage. Prometheus exposes per-group metrics for the last evaluation duration and for missed iterations; those are the numbers to watch when a rule set grows. The fixes are to split the group so independent rules run in parallel groups, to lengthen that group's `interval`, or to make the expensive expression cheaper. ## Naming, and the traps The documented convention is `level:metric:operations`: - **level** — the aggregation level, usually the labels that survive, such as `job` or `cluster`. - **metric** — the metric name being aggregated. - **operations** — the operations applied, most recent first, such as `rate5m` or `sum`. Colons appear only in recorded series names — instrumented metric names never use them — so a colon in a series name is an immediate signal that you are looking at a rule output rather than something an application exposed. Keeping to the convention also means a reader can tell from the name alone which labels are still present, which matters when an alert built on a recorded series can no longer identify the failing instance because the recording rule aggregated `instance` away. Two further traps: - **There is no backfill.** A recorded series exists only from the moment the rule was loaded. Adding a rule today and then asking for last quarter's numbers returns nothing, so create the rules that feed a long-horizon report well before you need the report. - **Recorded series cost storage.** Each one is a new series in the same store you were trying to protect. Recording every intermediate step of an expression multiplies series count; record the aggregates that are actually reused.