skip to content

Retries & Dead Letters

A consumer acks or retries each message, and messages that use up their retries move to a dead-letter queue. Interviewers ask how one bad message stops re-running a whole batch.

on this pageshow

explore

questions

5

In a Cloudflare Queues email consumer whose provider sometimes returns 429, how do message.ack() and message.retry() keep one failure from re-running the batch?

level: middleimportance: must knowfreq 45%

answer

  1. decide each message, then return
  2. negative acknowledgement
  3. retry with delaySeconds
  4. first call wins
  5. attempts drives backoff

basics

~20 s

Calling message.ack() after each successful send settles that message for good, and message.retry({ delaySeconds }) puts only the rate-limited one back for later, so no thrown error fails the batch and one failure no longer re-sends the emails beside it.

solid answer

~40 s

Without explicit calls a thrown error retries the whole batch, re-sending every email already delivered. So the consumer settles each message itself: `msg.ack()` right after a successful send marks it delivered whatever happens later, and on a 429 `msg.retry({ delaySeconds })` puts just that message back, delayed (up to 24 hours) by a backoff computed from `msg.attempts`. Nothing throws, so the other messages are unaffected. `batch.ackAll()` and `batch.retryAll()` do the same for every message. The first call on a message wins: `retry()` after `ack()` is ignored, and a later batch-level call does not override a per-message one. Retried messages still count toward `max_retries` (3 by default).

code

javascript · 30 lines
javascript
export default {
  async queue(batch, env) {
    for (const msg of batch.messages) {
      const email = msg.body;
      try {
        if (typeof email?.to !== "string") {
          // Malformed: park it on the dead-letter queue now instead of spending retries on it.
          await env.EMAILS_DLQ.send({ reason: "malformed", body: email });
          msg.ack();
          continue;
        }
        const res = await fetch(env.EMAIL_API_URL, {
          method: "POST",
          headers: { "content-type": "application/json", "idempotency-key": email.sendId },
          body: JSON.stringify(email),
        });
        if (res.status === 429) {
          // Rate-limited: back off this message only, longer on each attempt.
          msg.retry({ delaySeconds: Math.min(60 * 2 ** (msg.attempts - 1), 3600) });
        } else if (res.ok) {
          msg.ack();
        } else {
          msg.retry();
        }
      } catch {
        msg.retry();
      }
    }
  },
};

go deeper

for a junior

Name the four calls, ack(), retry(), ackAll() and retryAll(), and say which act on one message and which on the batch.

for a middle

Explain negative acknowledgement, the delaySeconds option with its 24-hour ceiling, a backoff from attempts, and the first-call-wins precedence rules.

for a senior

Write the loop so no error escapes it, ack right after each side effect, and note that at-least-once delivery still needs an idempotency key at the provider.

for a principal

Make per-message settlement the house pattern for any consumer with per-message side effects, and reserve the implicit batch outcome for genuinely all-or-nothing work.

A **Cloudflare Queues** consumer Worker receives messages in batches through its `queue(batch, env, ctx)` handler. By default the batch succeeds or fails as one unit: a normal return acknowledges every message, and a thrown error retries every message. For a consumer that sends one email per message, through a provider that sometimes answers **HTTP 429 Too Many Requests**, that default is wrong twice over: one rate-limited send would re-send every email already delivered in the batch, and the retry would come straight back at a provider that just asked you to slow down. The per-message API fixes both. ## The four calls | Call | Applies to | Effect | |---|---|---| | `message.ack()` | one message | Marks it delivered, whatever the handler does afterwards | | `message.retry(options?)` | one message | Marks it for redelivery in a later batch, without throwing | | `batch.ackAll()` | every message | Same as a successful return, even if the handler later throws | | `batch.retryAll(options?)` | every message | Marks the whole batch for redelivery, without throwing | `retry()` is what the docs call a **negative acknowledgement**: it puts one message back while the loop carries on with the rest. Messages that get neither call are settled by the batch outcome, so they are acknowledged when the handler returns normally. ## Delaying a retry Both retry calls accept `QueueRetryOptions`, which has a single field, `delaySeconds`. A delayed message is not redelivered until that many seconds have passed, up to **24 hours**. This is the tool for a 429: rather than hammering the provider, back the one message off. Each message carries an `attempts` count that starts at 1, so a backoff can grow with it, for example `Math.min(60 * 2 ** (msg.attempts - 1), 3600)` for 60, 120, 240 seconds and so on, capped at an hour. The consumer can also set a default delay for every retried message with `retry_delay` in its Wrangler config (or `--retry-delay-secs` on `wrangler queues consumer add`); a per-message `delaySeconds` overrides that default. ## Precedence: the first call wins The calls do not stack. The rules in the Queues docs: - After `ack()` on a message, later `ack()` or `retry()` calls on it are ignored. - After `retry()` on a message, a later `ack()` is ignored. - A per-message call made first is not overridden by a later `ackAll()` or `retryAll()`; the batch-level call applies only to the messages without one. The runtime logs a warning when it ignores a call, for example a `retry()` after `ackAll()`. In practice: decide each message's fate once, then let the batch default cover the rest. ## Putting it together for the email consumer 1. Validate the message. A malformed one will never succeed, so retrying it only burns attempts; send it to the dead-letter queue through a producer binding and `ack()` it. 2. Call the provider. On success, `ack()` at once, so a later failure in the same batch does not re-send this email. 3. On a 429, `retry({ delaySeconds })` with a backoff from `attempts`. 4. On any other error, `retry()` and let `max_retries` (3 by default) and the consumer's `dead_letter_queue` bound it. The code example for this question does exactly that. Two properties make it safe: - **No throw escapes the loop.** Every outcome is an explicit call, so one bad message never fails the others. - **Retries still count.** A message marked with `retry()` comes back with a higher `attempts` value and, once it has used up `max_retries`, goes to the dead-letter queue (or is deleted, without one) like a message that failed by throwing. ## What per-message acks do not solve Queues delivers **at least once**, so an email can still be sent twice, for example if the Worker is cut off after the provider accepted the send but before `ack()` was recorded. Per-message calls shrink that window from "the whole batch" to "one message"; they do not close it. The usual answer is to generate a unique ID when the message is produced and pass it to the provider as its idempotency key, so the provider rejects the duplicate. Designing that key in general is a question for the material on idempotent consumers; here the point is that `ack()` is a delivery decision, not a guarantee of exactly-once side effects. Finally, a caution about scale: explicit retries do not count as failed invocations, so they do not make Queues scale the consumer down. If the provider keeps returning 429 under a large backlog, the lever is a cap on consumer concurrency, not more retries.

  • What happens if a Queues consumer calls msg.ack() and then msg.retry() on the same message?
    The `retry()` is ignored and the runtime logs a warning: the first call on a message wins. The same holds the other way round, and a per-message call made first is not overridden by a later `batch.ackAll()` or `batch.retryAll()`.
  • Why retry a 429 with delaySeconds instead of letting the handler throw?
    A throw fails the whole batch, so every email already sent in it is sent again, and the retry comes back without a deliberate pause. `retry({ delaySeconds })` touches only the rate-limited message and gives the provider room. Explicit retries also do not count as failed invocations for consumer autoscaling.
  • How do retry_delay and a per-message delaySeconds interact?
    `retry_delay` on the consumer is the default delay for any retried message, whether it failed by a throw or by `retry()`. A `delaySeconds` passed to `retry()` or `retryAll()` takes precedence over it for those messages.

A mail sorter handling a sack of letters stamps each one delivered or return-later as it goes; one smudged address does not send the whole sack back to the depot.

  • message.retry() throws, so the rest of the batch stops processing.
  • batch.ackAll() after msg.retry() acknowledges the retried message as well.
  • Calling msg.ack() after msg.retry() cancels the pending retry.
  • A retry delay cannot be longer than the 60-second batch timeout.
  • Per-message ack() guarantees each email is sent exactly once.
open as a page

In a Cloudflare Queues consumer Worker that never calls ack(), when is a delivered batch acknowledged, and what makes it retry?

level: juniorimportance: should knowfreq 35%

basics

~20 s

A Queues batch is acknowledged implicitly once queue() returns, its promise resolves and every ctx.waitUntil() promise resolves. If queue() throws or any of those promises rejects, the whole batch fails and every message in it is retried.

open as a page

In Cloudflare Queues, what happens to a message that uses up its max_retries, with and without a dead_letter_queue configured?

level: middleimportance: should knowfreq 40%

basics

~20 s

Once a Queues message has used up max_retries (3 by default), it is deleted permanently unless the consumer sets dead_letter_queue, in which case it is written to that queue, which needs its own consumer to be processed.

open as a page

Why does a Cloudflare Queues email consumer keep hitting provider 429s during a backlog even though it retries them with delaySeconds, and how do you cap it?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Queues autoscales push consumers on backlog growth, and explicit retry() calls do not count as failed invocations, so concurrency keeps rising and more batches hit the provider at once. Set max_concurrency to what the provider can absorb.

open as a page

Why does one malformed message make a Cloudflare Queues email consumer send some welcome emails four times and then dead-letter its whole batch, and how do you fix it?

level: seniorimportance: should knowfreq 35%

basics

~20 s

A thrown error fails the whole Queues batch, so emails before the bad message are re-sent on each of four deliveries (default max_retries 3) and the batch exhausts its retries together. Fix it with per-message ack() and retry().

open as a page