In a Competing Consumers setup, a single message keeps causing a processing exception no matter which consumer handles it, and it keeps getting redelivered — starving the queue for other work. How do you design around this 'poison message' problem?

## What makes a message poison A **poison message** is any message that a consumer will never be able to successfully process, no matter how many times or which specific instance retries it — as opposed to a **transient failure** (a downstream API being briefly unavailable, a database connection blip) where a retry has a real chance of succeeding. The distinguishing property is that the failure is deterministic with respect to the message's content or the code path it triggers. Each of these will fail identically on every attempt, on every instance, forever: - a malformed JSON payload that fails to parse - a reference to a foreign-key ID that was since deleted - an edge case that triggers a genuine bug in the consumer Because the visibility-timeout/redelivery mechanism that makes Competing Consumers resilient to transient failures cannot distinguish "this will succeed if I just try again" from "this can never succeed," it treats every unacknowledged message the same way — it goes back on the queue for redelivery. Without an additional mechanism, a poison message enters an infinite retry loop. ## Why it matters This matters for two concrete reasons. 1. **First, it wastes consumer capacity.** Every redelivery consumes a worker's attention for the duration of the doomed processing attempt (including whatever time it takes to fail — parsing, partial work, the exception itself), capacity that could have gone to genuinely processable messages. Under load, a handful of poison messages cycling through the pool can measurably reduce effective throughput. 2. **Second, and more severely**, on a strict FIFO or single-partition queue where order matters, a poison message sitting at the head of the line can block every message behind it from ever being reached, since the consumer(s) assigned to that partition keep retrying the blocking message instead of moving on — this is sometimes called **head-of-line blocking**, and it turns one bad message into a total outage for that ordered stream, not just a wasted retry. ## The standard fix: a dead-letter queue The standard, broadly adopted fix is a **maximum-receive-count** (or maximum-redelivery-count) policy paired with a **dead-letter queue (DLQ)**. The broker (SQS, RabbitMQ with a policy, Azure Service Bus, etc.) tracks how many times each message has been delivered without being acknowledged. Once that count crosses a configured threshold — say, 5 attempts — instead of making the message visible again on the main queue, the broker automatically moves it to a separate DLQ. This achieves two things simultaneously: - the main queue's healthy throughput is no longer affected by that message (it stops competing for consumer attention and, on ordered queues, stops blocking subsequent messages); - the message itself is preserved rather than silently dropped, sitting in the DLQ for inspection. From there, typical practice is to alert on DLQ depth (a growing DLQ usually signals either a data-quality problem in producers or a real bug in the consumer), inspect the failed payloads to diagnose the root cause, and — once fixed — either manually replay the messages back onto the main queue or build an automated redrive pipeline for the common case. ## Choosing the threshold There's a real trade-off in how the max-receive-count threshold is chosen. | Threshold choice | What it costs you | |---|---| | Set it too low | legitimate transient failures (a downstream dependency that's flaky for a minute) get prematurely dead-lettered before a retry would have actually succeeded, turning a recoverable blip into manual intervention work | | Set it too high | a genuine poison message wastes more retries — and, on an ordered queue, blocks the line longer — before it's finally isolated | A common refinement is to distinguish failure types explicitly in the consumer: catch known-transient exceptions (a timeout, a 503) and let those retry normally up to the threshold, but catch known-permanent exceptions (a deserialization failure, a validation error on the payload itself) and route straight to the DLQ (or a "rejected" queue) on the first attempt, since retrying a guaranteed failure is pure waste — this requires the consumer to actively classify failures rather than treating every exception identically. ## Where it shows up A concrete, well-documented real-world instance: AWS SQS's redrive policy lets you configure a maximum receive count on a source queue together with a target DLQ, and this is the officially recommended pattern in AWS's own architecture guidance specifically to prevent poison messages from looping indefinitely; monitoring alarms on DLQ message count are the standard operational signal that something in the pipeline needs human attention. RabbitMQ achieves the equivalent via a queue's dead-letter-exchange policy combined with an explicit negative-acknowledgment (reject without requeue), routing repeatedly-rejected messages to a designated dead-letter exchange rather than looping them forever.