Explain delta versus cumulative aggregation temporality in OpenTelemetry metrics: what each data point carries, how a consumer detects a counter reset, and how the choice interacts with a scrape-based versus a push-based backend.

## The field and what it governs In OTLP, Sum and Histogram data points carry an `aggregation_temporality` field with values `DELTA` or `CUMULATIVE` (plus an unspecified value that consumers should treat as an error). Gauges have no temporality at all — a gauge reports a current value, and there is nothing to accumulate. Every point also carries two timestamps, `start_time_unix_nano` and `time_unix_nano`, and their meaning changes with temporality: - **Cumulative**: the value is everything accumulated from `start_time` to `time`. `start_time` stays constant across the whole life of the series. - **Delta**: the value covers only `start_time`→`time`, and each point's `start_time` equals the previous point's `time`. The windows tile end-to-end without gaps or overlap. ## Cumulative: stateful producer, resilient transport A cumulative series is a running total. Its great property is **idempotent recovery**: drop a point in transit and the next one still carries the missing increments, so a network blip costs resolution, not data. That makes cumulative naturally suited to a pull model, where the collector may scrape at irregular intervals or miss a scrape entirely. Its costs are producer-side state and reset ambiguity. The producer must remember every series' running total for as long as the series exists, so a series with many attribute combinations pins memory — and a series that stops being reported must eventually be forgotten, at which point resuming it looks like something new. When the process restarts, the total drops back toward zero: a consumer computing a rate must **detect the reset** and not report a huge negative rate. Detection uses two signals — the value decreased relative to the previous point, or `start_time` changed — and the second is the reliable one, which is why start timestamps are not optional decoration. ## Delta: stateless producer, fragile transport A delta series reports what happened in an interval and then forgets. The producer needs no long-lived accumulation, so memory is bounded by the series active in the current window and idle series simply stop being reported without any "is it a reset?" question. This suits environments with churn — short-lived workloads, functions, or very high attribute turnover — and it suits push backends that sum on the server side. The cost is that **every point is load-bearing**. Lose one and the total is permanently short by that amount; there is no later point that repairs it. Duplicate one — a retry that the backend does not de-duplicate — and the total is permanently over. That puts real weight on delivery guarantees and on the backend's de-duplication behaviour. Delta also makes a consumer's job of answering "what is the total since deploy?" a summation over time rather than a subtraction of two points. ## Conversion, and where it happens Cumulative → delta is straightforward: subtract consecutive points, handling resets. Delta → cumulative requires the converter to hold the accumulated state — exactly the state the producer avoided — so it moves the memory cost rather than removing it. This conversion is a normal function of a telemetry pipeline component when a producer and a backend disagree, and it is worth knowing that whoever converts pays. ## Backend fit **Scrape/pull-based systems** are built around cumulative counters: a scrape is a point observation of a running total, resets are detected by the query engine, and a missed scrape is harmless. Feeding them delta data requires an intermediate that accumulates. **Push-based hosted backends** frequently prefer delta, because the server aggregates across many senders and does not want each sender holding per-series totals — and because with ephemeral senders, cumulative series that vanish and reappear produce constant false resets. OpenTelemetry lets you choose: the OTLP metric exporter exposes a temporality preference (commonly configured as `cumulative`, `delta`, or a `lowmemory` option that mixes — typically delta for synchronous instruments while leaving asynchronous ones cumulative). The consequential rule is **do not mix temporalities within one series**: half the points meaning "since start" and half meaning "since last" produces numbers that are not wrong so much as meaningless, and backends generally cannot detect it. ## Practical failure signatures - Sawtooth graphs where a total repeatedly falls to zero: cumulative series restarting, either from real process restarts or from series being forgotten and re-created. - Enormous negative or spike rates at deploy time: a consumer failing to detect resets, or start timestamps not being set correctly. - Under-counted totals correlated with delivery problems: delta with lost points. - Steadily growing producer memory with flat traffic: cumulative with unbounded attribute cardinality — the temporality is not the bug, it is the amplifier. ## How to answer State the two definitions with their timestamp semantics, then the trade-off in one line each — cumulative buys transport resilience with producer state and reset handling; delta buys statelessness with sensitivity to loss and duplication — then match to backend model, and finish on the mixing rule and the memory-versus-cardinality interaction. That sequence is what separates a memorised definition from having operated it.