skip to content

In Redis, how do you use TS.CREATERULE to keep raw time series samples for a day and a downsampled series for a year?

level: middleimportance: should knowfreq 27%

answer

  1. the destination comes first
  2. retention lives on each key
  3. only samples after the rule
  4. the open bucket waits
  5. rollups fan out, never chain

basics

~20 s

Create a destination series first, with its own RETENTION, then run TS.CREATERULE source dest AGGREGATION avg 60000. Give raw a day's retention and the destination a year's. Only samples added after the rule are aggregated.

solid answer

~50 s

You keep two series. `TS.CREATE` the raw key with `RETENTION 86400000` and a destination with `RETENTION 31536000000`, then run `TS.CREATERULE raw dest AGGREGATION avg 60000`. The destination must exist first, or the command returns an error, and it does not inherit the source's retention. As samples arrive, Redis folds them into the current bucket and writes one sample to the destination, stamped with the bucket's start time, when a later sample opens the next bucket. `TS.RANGE` therefore hides the open bucket unless you pass `LATEST`. A rule only processes samples added after it exists, so it does not backfill. Each rule holds one aggregator, so min, avg and max need three destinations. Rules cannot chain, since a destination cannot be a source, so every rollup hangs off the raw key. In Redis Cluster, source and destination need a shared hash tag or compaction may fail silently.

code

plaintext · 22 lines
plaintext
TS.CREATE temp:{dev7} RETENTION 86400000 LABELS device 7 res raw
OK
TS.CREATE temp:{dev7}:1m RETENTION 31536000000 LABELS device 7 res 1m
OK
TS.CREATERULE temp:{dev7} temp:{dev7}:1m AGGREGATION avg 60000
OK
TS.ADD temp:{dev7} 60000 20
(integer) 60000
TS.ADD temp:{dev7} 90000 22
(integer) 90000
TS.RANGE temp:{dev7}:1m 0 200000
(empty array)
TS.ADD temp:{dev7} 120000 24
(integer) 120000
TS.RANGE temp:{dev7}:1m 0 200000
1) 1) (integer) 60000
   2) 21
TS.RANGE temp:{dev7}:1m 0 200000 LATEST
1) 1) (integer) 60000
   2) 21
2) 1) (integer) 120000
   2) 24

go deeper

for a junior

Recall that a rule copies aggregates from a source series into a destination series, and that you must create the destination with TS.CREATE before TS.CREATERULE.

for a middle

Explain when a bucket appears: only once a later bucket opens, stamped with its start time, with LATEST for the open bucket, and that retention is set separately on each key.

for a senior

Design around the limits: one aggregator per rule, no chaining, no backfill, O(M) writes, and a hash tag so source and destination share a slot in Redis Cluster.

for a principal

Choose rollups as a standing contract, because a statistic you did not compute before the raw data aged out is gone, and every extra rule costs on each write.

## What compaction is A **compaction rule** tells Redis to compute an aggregate incrementally as samples arrive and to store the result in a second time series. The source series is left unchanged. For a one-sample-a-second fleet, `RETENTION` on each key bounds that key's size, and the rule keeps a coarser history that outlives the raw data. At one sample a second, a device writes 86,400 raw samples a day and 1,440 one-minute buckets. ## Setting it up The order is strict: 1. `TS.CREATE` the raw series with `RETENTION 86400000`. 2. `TS.CREATE` the destination with `RETENTION 31536000000`. 3. `TS.CREATERULE source dest AGGREGATION avg 60000`. The arguments are the aggregator and `bucketDuration` in milliseconds, plus an optional `alignTimestamp`. Alignment shifts the bucket grid, so a 24-hour bucket can run from 06:00 to 06:00 instead of from midnight. The aggregator matters too: `avg` weights every sample equally, while `twa` computes a time-weighted average over the bucket, which suits devices reporting at uneven intervals. `TS.CREATERULE` replies with an error if: - the source or destination key does not exist - the source is already a destination of a rule - the destination is already a source or a destination of a rule - source and destination are the same key - more than one aggregator is given ## What each rule cannot do The error list implies three design limits. **One aggregator per rule.** `TS.RANGE` accepts `min,avg,max` in one query, but `TS.CREATERULE` takes exactly one. A dashboard that needs all three needs three destinations and three rules. **No chaining.** You cannot aggregate a one-minute series into a one-hour series, because a destination cannot be a source. Every rollup is a separate rule on the raw key, and each adds work: `TS.ADD` is O(M), where M is the number of rules on the key. **No backfill.** Only samples added after the rule exists are aggregated. Samples already in the raw series never reach the destination through the rule. Running `TS.CREATERULE` with a non-empty destination may also leave raw and compacted data inconsistent, as may writing to a destination directly, because compaction can override such samples. ## When a bucket appears The newest bucket is held back. Redis writes a compacted sample only when a source sample lands in a later bucket, on the reasoning that an aggregate is not final until its period is over. This produces behaviour that looks like a bug until you know it: | What you did | What the destination shows | |---|---| | Added samples inside the first bucket | Nothing | | Added a sample in the next bucket | The first bucket, stamped with its start time | | Added nothing during a whole bucket | No sample for that bucket | | Read with `TS.RANGE ... LATEST` | Closed buckets plus the open, possibly partial, one | The last row is what a live dashboard wants. `LATEST` applies to compaction series and is ignored elsewhere. Earlier buckets are not sealed either. If you add or delete raw samples in an older bucket, the rule updates that bucket's compacted value, working from the raw samples still in retention. That interaction with retention causes real production bugs, so size raw retention with late arrivals in mind. ## Retention is per key The destination has its own retention. Created with no `RETENTION`, it takes `ts-retention-policy`, which defaults to 0 and never expires. Nothing links the two values, so a one-day raw series and a one-year destination are ordinary settings on two keys. Deleting the destination also deletes the rule. ## Automatic rules The `ts-compaction-policy` parameter creates rules for you, written like `avg:1M:365d`, for keys created by `TS.ADD`, `TS.INCRBY` or `TS.DECRBY`. It does not touch keys made with `TS.CREATE`, which is why manually created destinations do not sprout rules of their own. Generated destinations get names like `key_AVG_60000`, plus `aggregation` and `time_bucket` labels. ## Redis Cluster A rule writes to its destination during the source's command, so both keys must share a hash slot. Use a hash tag, as in `temp:{dev7}` and `temp:{dev7}:1m`. The docs warn that without one, Redis may fail to compact the data with no error message. The same warning applies to `ts-compaction-policy`, which requires hash tags in every series name.

  • How do you add a rollup to a Redis time series that already holds a day of raw samples?
    Create the destination and the rule, and accept that the rule only sees later samples. If the older history matters, query it with `TS.RANGE` and `AGGREGATION` and load it into the destination before creating the rule. The docs warn that a non-empty destination may end up inconsistent with raw data, so reconcile any overlap around the cut-over.
  • Why might a Redis Cluster deployment show empty compaction series with no errors?
    The source and destination probably hash to different slots. The docs warn that Redis may then fail to compact without displaying any error message. Give both keys the same hash tag, such as `temp:{dev7}` and `temp:{dev7}:1m`, so they share a slot. With `ts-compaction-policy` set, every series name needs a hash tag.

A compaction rule works like a monthly summary of a daily log. The summary line is written once the month is over, and it stays after the old daily pages are thrown away. An entry backdated into a past month can only be summarised from the pages still on file.

  • TS.CREATERULE creates the destination series if it is missing.
  • A new compaction rule backfills the samples already in the source series.
  • The destination series inherits the retention of its source series.
  • You can chain rollups, feeding one-minute buckets into one-hour buckets.
  • A compaction rule runs on a background timer at the end of each bucket.
  • One rule can store min, avg and max in a single destination series.