A worker pops a job from a Redis list with LPOP and crashes while processing it. Describe how the LMOVE and BLMOVE commands let you build a queue that survives this, and what that design still leaves you to handle yourself.

## The failure the pattern fixes A naive Redis queue is RPUSH to produce and BLPOP to consume. The flaw is that popping is destructive and unacknowledged: once the reply leaves the server the element exists nowhere. If the worker dies, the network drops the reply, or the handler throws after receiving it, the job is gone with no trace. That is at-most-once delivery, fine for disposable work like cache warm-ups and unacceptable for anything a user paid for. ## The reliable-queue construction LMOVE source destination wherefrom whereto atomically removes an element from one end of the source list and pushes it onto one end of the destination. Atomically matters: there is no window in which the element exists in neither list nor in both. BLMOVE adds the blocking wait, so the consumer loop is a single command. The standard loop is: 1. job = BLMOVE jobs processing:w7 LEFT RIGHT 5 — take from the head of the shared queue, append to this worker's private in-flight list. 2. Process the job. 3. LREM processing:w7 1 job — acknowledge by removing exactly one matching entry. Use a per-worker processing list rather than one shared one. With a shared list you cannot tell which entries belonged to a worker that died, so recovery has to guess; with per-worker lists the recovery rule is simply 'everything in the list of a worker known to be dead'. ## What the crash looks like now If the worker dies after step 1, the job remains in processing:w7 forever and nothing in Redis notices. The recovery machinery is entirely yours to build, and describing it is what separates a senior answer from a textbook one: - Liveness: each worker maintains a heartbeat such as SET worker:w7:alive 1 EX 30, refreshed periodically. A reaper looks for processing:* lists whose worker heartbeat is missing. - Requeue: for each stranded entry, LMOVE processing:w7 jobs RIGHT LEFT to push it back for another attempt, or move it to a dead-letter list after too many tries. - Attempt counting: bare list entries carry no metadata, so the payload itself should include an id and an attempt counter, or a parallel hash keyed by job id should track attempts. Without this a poison job loops forever, burning a worker each pass. - Idempotency: because a job can be redelivered after a crash that happened late in processing, handlers must tolerate running twice. Deduplicate on a job id, or make effects naturally idempotent. ## Costs and caveats LREM is O(N) in the length of the processing list, which is fine when that list holds one or a few in-flight jobs per worker and terrible if it accumulates thousands of stranded entries. Monitor LLEN on processing lists; growth means the reaper is not keeping up. Acknowledging by value also makes duplicate identical payloads ambiguous — LREM removes some matching entry, not necessarily yours. Include a unique id in every payload to keep entries distinct. The move and the acknowledgement are separate round trips, so the guarantee is at-least-once, never exactly-once. And because Redis replication is asynchronous, an element moved on a primary that fails over before the replica caught up can be redelivered or lost — the pattern makes worker crashes survivable, not node failovers. ## When to stop If you find yourself building heartbeats, attempt counters, dead-letter lists and a reaper, you are reimplementing consumer groups. Redis Streams provide pending-entry tracking and acknowledgement natively and are usually the better structure for a durable work queue. The list pattern earns its place when the workload is simple, volume is modest, and you want no extra concepts.