skip to content

How do you page through a large Redis Stream with XRANGE without re-reading or skipping entries at the page boundaries?

level: middleimportance: should knowfreq 35%

answer

  1. XRANGE boundaries inclusive → boundary duplicates
  2. `(id` exclusive prefix, Redis 6.2+
  3. Or bump the seq: `1526-3` → `1526-4`
  4. Start ID need not exist — gaps are fine
  5. Always COUNT; stop on a short page

basics

~20 s

Ask for COUNT+ entries from a start ID, then make the next page start just after the last ID you got. Since XRANGE boundaries are inclusive, either use the exclusive prefix — XRANGE key (lastId + COUNT n — or increment the sequence part of that ID yourself.

solid answer

~50 s

`XRANGE` boundaries are **inclusive on both ends**, so naively reusing the last returned ID as the next page's start re-delivers that entry. Two correct approaches: 1. **Exclusive boundary (Redis 6.2+)** — prefix the ID with `(`: ``` XRANGE s - + COUNT 100 # page 1, last id = 1526-3 XRANGE s (1526-3 + COUNT 100 # page 2, starts strictly after ``` 2. **Increment the sequence yourself** — IDs are `<ms>-<seq>`, so the successor of `1526-3` is `1526-4`. This works on any version, and is safe because IDs are strictly increasing and never reused: even if `1526-4` never existed, the range simply starts there. For descending paging with `XREVRANGE`, the same rule applies in reverse: `XREVRANGE s (1526-3 - COUNT 100`, or decrement the sequence. **Always pass COUNT.** `XRANGE s - +` on a multi-million-entry stream builds one enormous reply, which occupies the server's single execution thread and can stall every other client. Terminate the loop when a page comes back with fewer than COUNT entries.

code

text · 11 lines
text
> XRANGE s - + COUNT 100
... last id in page = 1526919030474-3

# Redis 6.2+: exclusive boundary
> XRANGE s (1526919030474-3 + COUNT 100

# any version: increment the sequence
> XRANGE s 1526919030474-4 + COUNT 100

# the buggy version — re-delivers 1526919030474-3
> XRANGE s 1526919030474-3 + COUNT 100

go deeper

for a junior

Know that XRANGE includes both boundaries, so the next page must start after the last ID you saw, and that COUNT limits page size.

for a middle

Show both correct cursors — the ( exclusive prefix and incrementing the sequence — and stop the loop on a short page.

for a senior

Add the operational reasoning: single-threaded server means unbounded replies stall everything, cursors are IDs so gaps and trimming are fine, and pin the end boundary if you need a stable snapshot of a live stream.

for a principal

Tie it to the delivery contract: with no server-side acknowledgement, where you persist the cursor decides at-least-once versus at-most-once, and retention must be sized against worst-case reader lag.

## Why paging needs care here A Redis Stream is stored as a radix tree of macro-nodes, so `XRANGE` is efficient — O(log N) to locate the start of the range plus O(M) for the M entries actually returned. The danger is not the lookup, it is **how much you ask for in one call**: Redis executes commands on a single thread, and serializing a hundred thousand entries into one reply blocks everything else for the duration, and inflates the client output buffer. So paging is not an optimization, it is the correct way to read a large stream at all. The correctness trap is the boundary semantics. ## Inclusive boundaries ``` XRANGE key start end [COUNT n] ``` `start` and `end` are both **inclusive**. `XRANGE s 5-0 9-0` returns entries `5-0` through `9-0` inclusive. Therefore, if page 1 ended at `1526-3` and page 2 starts at `1526-3`, entry `1526-3` is processed twice. If your handler is not idempotent, that is a real bug — and it hides well, because it only shows up at page boundaries. ## Fix 1: the exclusive prefix Since Redis 6.2 both `XRANGE` and `XREVRANGE` accept `(` before a boundary ID to make it exclusive: ``` XRANGE s (1526919030474-3 + COUNT 100 ``` This is the clearest expression of intent. Note that `(-` and `(+` are errors: the sentinels have no meaningful exclusive form. ## Fix 2: increment the ID Works on every version, and is what most client libraries did before 6.2. Given `<ms>-<seq>`, the next possible ID is `<ms>-<seq+1>`. Because the sequence is a 64-bit counter you will not overflow in practice; if you wanted to be pedantic, the successor of `<ms>-<2^64-1>` is `<ms+1>-0`. Crucially, the start of a range **does not need to exist**. Ranges are positional over an ordered structure, so starting at a gap simply begins at the next entry that does exist. Streams are full of gaps after `XDEL` and trimming, so any paging scheme that assumes the cursor ID is still present is wrong. This is also why you should store the last-seen ID as your cursor rather than an offset or an index — there is no index into a stream. ## Fix 3: reuse XREAD's exclusivity `XREAD COUNT 100 STREAMS s <lastId>` is already exclusive of `<lastId>`, with no prefix or arithmetic needed. Without `BLOCK` it returns immediately, so a catch-up loop over history is perfectly reasonable with `XREAD`, and many people use it precisely to avoid the off-by-one. The tradeoff: `XREAD` has no upper boundary, so you cannot page a bounded window ("just this hour") with it, and it returns nil rather than an empty array when nothing matches. ## Loop termination and drift Terminate when a page returns **fewer entries than COUNT**, which means you have reached the current end of the stream — not when it returns empty, which costs an extra round trip but is also correct. Be aware the stream is live: entries can be appended while you page. Ascending paging naturally picks up newcomers because they have larger IDs — usually desirable for a backfill-then-tail flow, but it means an ascending scan of a busy stream may never terminate. If you want a stable snapshot, fix the end boundary up front (`+` replaced by the current `last-generated-id` from `XINFO STREAM`, or by the current time in ms) and page towards it. Concurrently, entries **behind** your cursor may be trimmed away. That does not break paging — your cursor is an ID, not an offset — but it does mean a slow reader can miss data on an aggressively trimmed stream. If a reader must not miss entries, size the retention against the worst-case reader lag. ## Descending paging `XREVRANGE key end start [COUNT n]` takes the boundaries in reverse order. The idiom for the newest page is `XREVRANGE s + - COUNT 20`; the next (older) page is `XREVRANGE s (<oldestIdSeen> - COUNT 20`, or decrement the sequence. Descending paging over a live stream has the nice property that appends do not disturb pages you have already fetched, which is why UI feeds are usually built this way. ## Putting it together A typical backfill-then-follow worker: 1. `cursor = storedLastId or "-"`. 2. Loop `XRANGE key (cursor + COUNT 500`, process, persist `cursor = lastId of page`, until a short page. 3. Switch to `XREAD BLOCK 5000 STREAMS key cursor` for the live tail, still persisting the cursor after each batch. The cursor persistence point determines your delivery semantics: persist after processing for at-least-once (duplicates possible on crash), before processing for at-most-once (loss possible). Plain `XRANGE`/`XREAD` give you no server-side acknowledgement, so this choice is yours to make explicitly.

  • What if the cursor ID you page from has since been deleted or trimmed away?
    It does not matter. XRANGE boundaries are positions in an ordered structure, not lookups, so a start ID that no longer exists simply begins the range at the next surviving entry. This is why an ID makes a durable cursor even on a stream that is actively trimmed — but it also means a slow reader can silently skip past entries that were trimmed while it was behind.
  • Why not just call `XRANGE key - +` once and page in the client?
    That builds the entire reply on the server in one command. Redis executes commands on a single thread, so serializing millions of entries blocks every other client for the duration, and the reply can exhaust the client output buffer limit and get the connection killed. COUNT keeps each command short and the server responsive; it is the same discipline as using SCAN instead of KEYS.
  • How do you page a feed newest-first?
    Use `XREVRANGE key + - COUNT n` for the first page, then `XREVRANGE key (<oldestIdSeen> - COUNT n` for each older page, decrementing instead of incrementing. Descending paging has the practical advantage that new appends never shift pages you have already returned, which keeps a UI feed stable while it is being scrolled.

saying these in an interview costs you the question

  • Reusing the last returned ID as the next page's start and duplicating a boundary entry.
  • Calling XRANGE with `- +` and no COUNT on a large stream.
  • Assuming the cursor ID must still exist in the stream for the next page to work.
  • Treating a stream position as a numeric offset or index rather than an ID.
  • Expecting an ascending scan of a live, actively written stream to terminate on its own.

context

resolving links…