skip to content

Several worker processes need to share the entries of one Redis Stream so that each entry is initially handed to only one worker. How do you set that up with XGROUP CREATE and XREADGROUP, and what does the special ID `>` mean in XREADGROUP?

level: middleimportance: must knowfreq 55%

answer

  1. XGROUP CREATE key group $ | 0 | MKSTREAM
  2. BUSYGROUP = already exists, swallow it
  3. XREADGROUP GROUP g consumer ... STREAMS key >
  4. `>` = never-delivered; any ID = my own pending history
  5. stable consumer names, reading never deletes

basics

~20 s

Create the group once: XGROUP CREATE key group $ (use 0 to replay history, MKSTREAM if the stream may not exist). Each worker reads with XREADGROUP GROUP group consumer-name STREAMS key >. The > means entries never yet delivered to anyone in this group, so Redis gives each new entry to one consumer only.

solid answer

~50 s

A consumer group is server-side bookkeeping attached to a stream key. Create it once: `XGROUP CREATE mystream mygroup $ MKSTREAM`. `$` starts at the current end so only future entries are delivered; `0` replays everything already stored; `MKSTREAM` creates an empty stream if the key is missing; a second CREATE returns BUSYGROUP. Each worker then reads with a stable consumer name: `XREADGROUP GROUP mygroup worker-3 COUNT 10 BLOCK 5000 STREAMS mystream >`. Consumers are created implicitly on first read. `>` means "entries never delivered to any consumer in this group". Redis advances the group's `last-delivered-id`, returns those entries to that single caller, and records them in the group's Pending Entries List under that consumer name. Passing an explicit ID instead (normally `0`) switches to history mode: it returns that consumer's own already-delivered, unacknowledged entries, never new ones, and never blocks. Entries are not removed by reading; other groups on the same key consume the same entries independently.

code

text · 8 lines
text
# once at startup (ignore BUSYGROUP if it already exists)
XGROUP CREATE orders billing $ MKSTREAM

# each worker, with its own stable name
XREADGROUP GROUP billing worker-3 COUNT 10 BLOCK 5000 STREAMS orders >

# on restart: first drain what this worker was already holding
XREADGROUP GROUP billing worker-3 COUNT 10 STREAMS orders 0

go deeper

for a junior

Know the two commands and their shape: create the group once, then every worker calls XREADGROUP with its own consumer name and > to get new entries.

for a middle

Explain what $ vs 0 does at creation, what > vs an explicit ID does at read time, and that delivery state lives in the group while entries stay in the stream.

for a senior

Add the operational points: stable consumer names, idempotent creation swallowing BUSYGROUP, MKSTREAM ordering, BLOCK/COUNT tuning, and reading pending history with ID 0 on restart.

for a principal

Frame group topology as a design choice — one group per pipeline, partitioning by stream key rather than by consumer, replay strategy via XGROUP SETID, and how lag and trimming policy interact across multiple groups on one key.

## What a stream gives you without a group A Redis Stream is an append-only log under one key. Each entry has an ID of the form `<milliseconds>-<sequence>` (e.g. `1712345678901-0`) and a set of field/value pairs. Reading with XREAD is a fan-out read: every client that reads sees every entry, and each client must remember its own position. The server keeps no record of who read what and no record of whether anyone finished processing an entry. That is fine for tailing a log, but it does not let you run a pool of N interchangeable workers that split the load, and it gives you no way to notice that a worker died mid-job. A consumer group adds exactly those two things: a shared cursor and per-entry delivery tracking. ## Creating the group ``` XGROUP CREATE mystream mygroup $ MKSTREAM ``` - `mystream` is the stream key; a group belongs to one key. - `mygroup` is the group name. Multiple groups can exist on the same stream and each has its own independent cursor and tracking. - The third argument is the starting ID. `$` means "the current last ID", so the group only ever sees entries added after creation. `0` means "from the very beginning", so the group will replay everything still stored in the stream. You can also give any concrete ID. - `MKSTREAM` creates the stream as an empty key if it does not exist yet. Without it, creating a group on a missing key returns an error, which is a classic startup ordering bug when producers have not run yet. - Creating a group that already exists returns a `BUSYGROUP` error. Startup code normally issues the CREATE and swallows exactly that error, which makes the call idempotent. `XGROUP SETID` can later move the group's cursor (for a deliberate replay or a skip-ahead). `XGROUP DESTROY` removes the group and all its tracking. ## Reading as a consumer ``` XREADGROUP GROUP mygroup worker-3 COUNT 10 BLOCK 5000 STREAMS mystream > ``` - `GROUP mygroup worker-3` names the group and this worker's consumer name. Consumers are created implicitly on first read; there is no registration step (though `XGROUP CREATECONSUMER` exists if you want one explicitly). - `COUNT` caps how many entries come back in this call. - `BLOCK 5000` waits up to 5 seconds for new entries instead of returning empty; `BLOCK 0` waits forever. This blocks the calling client, not the server. - `STREAMS mystream >` is the key and the ID to read from. The consumer name matters operationally: it must be stable across restarts. If each process boots with a random name, the entries a previous incarnation was holding stay attached to a consumer name that will never come back, and you accumulate orphaned pending entries plus a growing consumer list. ## What `>` means, and what any other ID means `>` is a special ID that only exists for XREADGROUP. It means: give me entries that have never been delivered to *any* consumer of this group. Serving it does three things atomically: 1. returns the entries to this one caller, 2. advances the group's `last-delivered-id` past them, and 3. inserts them into the group's Pending Entries List (PEL), tagged with this consumer name, a delivery timestamp and a delivery counter. Because step 2 happens on the server under the single command execution, two workers issuing XREADGROUP at the same instant get disjoint sets of entries. That is the load-splitting property. Any other ID (in practice `0`, or `0-0`) switches the call into history mode. It returns entries from *this consumer's own* pending list with IDs greater than the one given, never any new entries, never blocks, and does not move `last-delivered-id`. That is what a worker uses on restart to pick up work it had already been handed but never finished. The optional `NOACK` flag skips the PEL entirely: the entry is delivered and forgotten, trading redelivery-on-failure for less bookkeeping. ## What the group tracks, and how to look at it `XINFO GROUPS mystream` shows each group's consumer count, pending count, `last-delivered-id`, and (since 7.0) `entries-read` and `lag`. `XINFO CONSUMERS mystream mygroup` shows per-consumer pending counts and idle time. `XINFO STREAM mystream` shows length and first/last IDs. Two separations are worth internalising. First, delivery state lives in the group, not in the entry: the same entry can be pending in group A, acknowledged in group B, and never seen by group C. Second, reading does not consume: entries remain in the stream until XDEL or trimming removes them, so stream length is governed by your MAXLEN/MINID policy, not by acknowledgements. ## The guarantee you actually get A fresh entry is handed to one consumer. It is not exactly-once: if that consumer dies before acknowledging, the entry stays pending and another consumer can take it over, so the end-to-end guarantee is at-least-once and handlers must tolerate seeing an entry twice.

  • You created the group with `$` but the stream already had a thousand entries and none are delivered. Why, and how do you fix it?
    `$` sets the group's starting cursor to the stream's current last ID, so everything written before creation is considered already past. Nothing is lost from the stream itself, only from this group's view. Recreate the group with `0`, or move the existing group's cursor with `XGROUP SETID mystream mygroup 0` to replay from the beginning.
  • What changes if two different consumer groups read the same stream key?
    Each group has its own `last-delivered-id` and its own pending list, so both groups see every entry independently; splitting happens only among consumers within a group. That is how you run, say, a billing pipeline and an analytics pipeline off one stream. The entries themselves are stored once, so memory cost is shared and trimming affects both.

The stream is a printed order queue on the wall; the group is a shift with a clipboard. > is "give me the next order nobody has taken"; an explicit ID is "re-read the orders already on my clipboard".

saying these in an interview costs you the question

  • Saying XREADGROUP deletes or consumes entries — reading never removes anything from the stream
  • Using a random or PID-based consumer name on every restart, orphaning that consumer's pending entries
  • Believing a group gives exactly-once delivery rather than at-least-once
  • Calling XGROUP CREATE on every startup without handling BUSYGROUP, and crashing the service
  • Thinking `>` means "from the beginning" or "latest ID" rather than "never delivered to this group"

context

resolving links…