Competing Consumers

## How it works The **Competing Consumers** pattern puts a pool of consumer instances in front of one shared queue, and lets the messaging infrastructure — not the application — decide which instance handles each message. Concretely: 1. A producer publishes a message onto a queue (an `SQS` queue, a `RabbitMQ` queue, a `JMS` destination, etc.). 2. Several consumer processes are all subscribed to or polling that same queue. 3. When a message arrives, the broker hands it to exactly one of the waiting consumers, typically whichever asks first or via round-robin/prefetch rules. 4. The other consumers do not see that message at all — they are "competing" for the next one. This is fundamentally different from **publish-subscribe**, where every subscriber gets a copy of every message; here, each message is consumed by exactly one instance, and increasing the pool size increases parallelism rather than duplication. ## Why the pattern exists The pattern exists to decouple the rate at which work arrives from the rate at which a single process can do that work, and to make that capacity elastic. A common scenario: a web application accepts image-upload requests and needs to generate thumbnails. - If thumbnailing happened **synchronously** in the request handler, a burst of uploads would slow down every user's response time, and CPU-heavy work would compete with request-serving. - Instead, the handler just enqueues a "generate thumbnail" message and returns immediately; a separate fleet of worker processes pulls from that queue and does the CPU-heavy work. - Because the workers are **stateless** with respect to each other, you can run one worker during quiet hours and fifty during a traffic spike — the queue naturally load-balances across however many workers exist, and no code change is needed to scale. - This also buys **fault tolerance**: if a worker crashes mid-message, a visibility/lock mechanism makes that message available again so a surviving worker picks it up, so no single worker instance is a point of failure for the whole pipeline. ## The trade-offs The trade-offs are real, though. 1. **First, ordering is generally lost.** If message A is enqueued before message B, there is no guarantee A finishes processing before B, since they may land on different consumers with different processing times. Anything that assumes strict FIFO semantics across the whole queue needs a different strategy (partitioning by key, a single-consumer-per-partition model like Kafka consumer groups, or accepting eventual/local ordering only). 2. **Second, most competing-consumers setups deliver at-least-once, not exactly-once.** A consumer that crashes after doing the work but before acknowledging it will see the message redelivered, so consumer logic must be idempotent (safe to run twice) or the system needs deduplication. 3. **Third, there's a cost/latency trade-off in how many consumers you run.** Too few and the queue backs up under load (rising latency, growing backlog); too many and you pay for idle capacity and may overwhelm downstream systems (a database, a third-party API) that the consumers all call into — the queue decouples the producer from the consumer, but the consumer fleet is still coupled to whatever it writes to. ## Failure modes in production Failure modes show up concretely in production. - A **poison message** — one that always throws an exception no matter which consumer processes it — can get redelivered endlessly, wasting worker cycles and potentially blocking a first-in-first-out queue behind it; the fix is a **dead-letter queue** that captures messages after N failed attempts. - **Uneven message cost** (some jobs take 10ms, others take 10 minutes) can starve the pool if a slow message monopolizes one consumer while others sit idle waiting for new work — this argues for finer-grained messages or a worker pool sized for the P99 job, not the average. - And if consumers acknowledge a message before finishing the work (to be "fast"), a crash mid-processing silently loses that message — the safe order is always **process-then-acknowledge**. ## Where it shows up A concrete real-world instance: AWS SQS standard queues are built exactly for this pattern — a single queue with an auto-scaling group of EC2 instances or a Lambda function with reserved concurrency, where CloudWatch scales the consumer count based on the approximate number of visible messages. RabbitMQ's own tutorial series calls this exact setup "Work Queues," explicitly demonstrating round-robin dispatch across multiple worker scripts consuming the same queue to parallelize CPU-heavy tasks.