In the competing consumers pattern, several consumer instances pull work from the same shared queue to scale out processing. What delivery-guarantee problems does this introduce that a single consumer wouldn't have, and how are they typically handled?

## The mechanism The mechanism is: one logical queue, N worker processes, each independently pulling (or being pushed) the next available message and racing to be the one that claims it. The queue's job is to hand each message to exactly one worker at a time. It typically does this with a **"visibility timeout"** or **"lease"** model: 1. When a worker takes a message, the queue makes it invisible to other pullers for a configured window (say, 30 seconds), betting that's enough time to finish the work. 2. The worker must explicitly acknowledge success before that window expires; if it does, the message is permanently deleted. 3. If the window expires with no ack — because the worker crashed, hung, or was simply slower than expected — the message becomes visible again and some other worker (or the same one, restarted) picks it up. This is the load-balancing mechanism in competing consumers: not a central dispatcher assigning work up front, but workers pulling on demand, which self-balances load because a fast or idle worker naturally grabs the next item sooner than a busy one. ## Why the pattern exists This pattern exists to solve horizontal scaling of processing capacity for a stream of independent, order-insensitive work items — background jobs, image resizing tasks, email sends, webhook deliveries. Unlike consumer groups over partitioned topics, there's no partitioning step and no ordering guarantee to preserve; you deliberately give that up in exchange for maximally even load distribution, since any idle worker can take any pending message rather than being pinned to a subset of the stream. It's the natural fit when work items are independent of each other and you just want "get through the backlog as fast as possible, using however many workers I currently have." ## The central trade-off The central trade-off is exactly the one the question calls out: giving multiple workers access to a shared pool introduces a **crash-recovery gap** that a single consumer never has. - With a single consumer, if it crashes mid-item, there's no ambiguity about who should retry — it's the only consumer, so on restart it just resumes. - With competing consumers, the system needs an explicit mechanism (visibility timeout, lease, or ack/nack protocol) to detect that the original worker died and hand the message to someone else, and that mechanism inherently can't be perfect: it has to guess a timeout long enough to cover normal processing time but short enough to recover promptly from a real crash, and workers that are simply slow — not dead — will have their message reassigned and processed twice. The unavoidable consequence is **at-least-once delivery**: you cannot get exactly-once without either a distributed transaction spanning the queue and every side effect the worker performs, or an idempotency mechanism, both of which cost engineering effort the pattern doesn't hand you for free. ## Failure modes Failure modes follow directly. - **Duplicate processing** happens when a worker finishes the real work — e.g., charges a card, sends an email — but crashes or the network drops before its ack reaches the queue; the message reappears and another worker redoes the side effect, so anything with an external, non-idempotent effect needs a dedup mechanism (idempotency keys, "already processed" markers) or you get double charges or duplicate emails. - **A poison message** — one that a worker can never successfully complete, e.g., malformed data that always throws — will otherwise cycle endlessly: worker takes it, crashes or throws, timeout expires, next worker takes it, same result, forever, quietly consuming worker capacity and cluttering logs. The standard fix is a bounded retry count tracked per message, with automatic routing to a **dead-letter queue** once exceeded, so a human or a separate remediation process handles it out of band rather than it blocking the main flow. - **Visibility-timeout mistuning** is a subtler failure: set it too short relative to real processing time under load, and you get a stampede of duplicate processing as messages get reassigned to new workers before the original one is actually done; set it too long, and a genuinely crashed worker's message sits invisible and unprocessed for that whole window, inflating latency during recovery. ## A worked example A concrete example: an image-processing pipeline pushes a "resize this uploaded photo" message per upload onto a shared queue, with 20 worker instances competing to pull from it. Each worker takes a message, resizes the image, uploads the result, then acks. If a worker is killed mid-resize (e.g., an out-of-memory kill), its message's visibility timeout expires after 60 seconds and another worker picks it up and redoes the resize — wasted but harmless work since resizing is naturally idempotent (same input produces the same output, and overwriting the output object is safe). If instead the queue were used for "charge this customer's card," the team would need to record a per-message idempotency key checked against the payment provider before charging, precisely because a duplicate redelivery there is not harmless.