Before a message ever reaches a dead-letter queue, how should a retry policy typically combine a backoff schedule with a max-delivery threshold, and why not just retry immediately and forever?

## How a retry policy computes the next attempt A retry policy tracks a per-message delivery attempt count and, on each failure, computes a delay before the next attempt using a backoff formula — commonly exponential: `base_delay * 2^attempt`, often capped at a maximum delay so it doesn't grow unbounded, and jittered by adding a random offset. Concretely: 1. Attempt 1 fails at `t=0`; the consumer waits ~1s (+jitter) before attempt 2. 2. Attempt 2 fails and waits ~2s (+jitter) before attempt 3, doubling each time. …until either the message succeeds or the attempt count reaches a configured **max-delivery threshold** (e.g. 5 or 8), at which point the message is routed to the DLQ instead of being retried again. ## Two very different kinds of failure This combination exists because failures split into two very different categories. | Category | What it looks like | What retrying achieves | |---|---|---| | **Transient failures** | a network blip, a downstream 503 during a deploy, database connection-pool exhaustion, rate limiting | likely to succeed if simply tried again after some delay | | **Permanent failures** | a malformed payload, a business-rule violation, a reference to an entity that will never exist | will never succeed no matter how many times or how long you wait | - **Backoff** exists to protect the downstream dependency: retrying instantly (or on a fixed short interval) from many consumer instances at once, right as a dependency is already struggling, is a retry storm that can prevent that dependency from ever recovering — this is exactly why many resilience toolkits pair backoff with circuit breakers. - **The threshold** exists because backoff alone doesn't stop retrying a permanent failure; it just makes the retries very slow forever, still consuming a delivery slot indefinitely and never surfacing the underlying problem to a human. ## The trade-off The trade-off sits between total end-to-end latency and how quickly poison messages get isolated. - An **aggressive threshold** (say, 2 attempts) isolates poison messages fast but risks routing genuinely-transient hiccups to the DLQ, generating noise and manual toil. - A **lenient threshold** (say, 20 attempts with long backoff) gives transient failures more chances to self-heal, but lets a true poison message occupy a delivery slot — or block an ordered partition — for a long time, and its total wait can stretch into hours if the backoff is exponential and uncapped. That's why production systems almost always cap the maximum per-attempt delay so backoff plateaus rather than growing indefinitely. ## Failure modes Common production failure modes: 1. **No jitter** causes synchronized retries across many consumer instances, creating load spikes at fixed intervals right when the downstream is trying to recover, worsening rather than easing an outage. 2. **Backoff misaligned with the broker's own redelivery mechanics** is another: in SQS, if the queue's visibility timeout is fixed and shorter than your intended app-level backoff, the message becomes visible for redelivery again before your delay logic meant it to, effectively defeating the backoff. 3. **Thresholds sized without reference to the dependency's actual outage profile** cause premature DLQ routing — if a downstream's typical blip lasts 90 seconds but the cumulative backoff exhausts the retry budget in 30, recoverable messages get dead-lettered on every minor deploy or scaling event. ## A worked example A worked example: a payment-processing consumer calling a third-party gateway that occasionally returns 429/503 under load. Using exponential backoff starting at 500ms, doubling up to a 30s cap, with full jitter, and a max-delivery threshold of 8, the vast majority of transient gateway hiccups (which typically self-resolve within seconds) succeed on retry 2 or 3. Only genuinely bad messages — an amount of -5, or a card ID that will never validate — survive all 8 attempts and land in the DLQ, at which point it's clearly a data problem worth a human's attention, not a load problem the system should have absorbed on its own.