Why does one malformed message make a Cloudflare Queues email consumer send some welcome emails four times and then dead-letter its whole batch, and how do you fix it?
answer
- one throw, one verdict
- first delivery plus three retries
- the batch runs out together
- ack after each send
basics
~20 sA thrown error fails the whole Queues batch, so emails before the bad message are re-sent on each of four deliveries (default max_retries 3) and the batch exhausts its retries together. Fix it with per-message ack() and retry().
solid answer
~40 sWithout explicit calls, a throw fails the whole batch, so Queues redelivers every message in it. On each delivery the loop re-sends the emails before the malformed message, throws again on it, and never reaches the ones after. With the default `max_retries` of 3 that is four deliveries, and because every message failed every time, they all run out together and land on the `dead_letter_queue`. The fix: validate first and dead-letter a malformed message yourself through a producer binding, `msg.ack()` right after each successful send, `msg.retry()` transient failures instead of throwing, and give the provider an idempotency key so a redelivery cannot send twice.
code
javascript · 15 linesexport default {
async queue(batch, env) {
for (const msg of batch.messages) {
const email = msg.body;
if (typeof email.to !== "string") {
throw new Error(`malformed email ${msg.id}`);
}
await fetch(env.EMAIL_API_URL, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(email),
});
}
},
};go deeper
Remember that a thrown error retries every message in the batch, and that the default is three retries, so four deliveries in all.
Trace the loop on each delivery: messages before the bad one are re-sent, messages after it never run, and all of them exhaust retries together.
Fix it with validation, a direct dead-letter send, ack after each side effect and retry instead of throw, plus an idempotency key, and know that local simulation ignores acks when the handler throws.
Turn the incident into a rule: consumers with per-message side effects never let an error escape the loop, and dead-letter replays are keyed so they cannot repeat a side effect.
This is a production diagnosis with one right explanation. A **Cloudflare Queues** consumer Worker sends one welcome email per message. A producer enqueues a message with no `to` field, the consumer throws on it, and the symptoms are: some recipients get the same email four times, others in the same batch get none, and then every message of that batch, good and bad, shows up on the dead-letter queue. ## Step 1: a throw fails the whole batch The consumer in the code example loops over `batch.messages`, calls the provider for each, and throws when a message has no recipient. With no explicit `ack()` or `retry()` calls, Queues settles the batch by its outcome: a normal return acknowledges every message, and a throw fails every message. Queues sees one invocation fail; it cannot know that the messages before the bad one were handled, so it redelivers all of them. ## Step 2: why some emails go out four times Every message in the failed batch comes back. On each delivery the loop starts again from the top: - Messages **before** the malformed one in the loop are sent again, every time. - The malformed message throws again, every time; it can never succeed. - Messages **after** it are never reached, so their emails are never sent. The consumer's `max_retries` defaults to **3**, so each message is delivered four times: the first delivery plus three retries. Queues does not guarantee ordering, so which messages fall before or after the bad one can change between deliveries, but any message the loop reaches before the throw is sent once per delivery. ## Step 3: why the whole batch is dead-lettered Retries are counted per message, and every message in the batch failed every time, because the batch failed. So all of them run out of retries together, and all of them are written to the `dead_letter_queue`, including messages whose emails were sent four times. Without a dead-letter queue they would all have been deleted. Two more costs come with it: each retry is billed as an extra read operation, and failed invocations weigh on how consumer concurrency scales. ## The fix Make each message's fate a per-message decision, so no error escapes the loop: ```javascript for (const msg of batch.messages) { if (typeof msg.body?.to !== "string") { await env.EMAILS_DLQ.send({ reason: "malformed", body: msg.body }); msg.ack(); continue; } try { await sendEmail(msg.body, env); msg.ack(); } catch { msg.retry(); } } ``` 1. **Validate first, and dead-letter what can never work.** A dead-letter queue is an ordinary queue, so a producer binding to it lets the consumer park the bad message at once instead of spending its retries. 2. **`ack()` each message right after its side effect.** An acknowledged message is not redelivered even if a later message in the batch fails. 3. **`retry()` transient failures** instead of throwing; they count toward `max_retries` and reach the dead-letter queue when exhausted. 4. **Pass a stable idempotency key to the provider.** Generate it when the message is produced. Delivery is at least once, so a crash between the provider accepting a send and the `ack()` can still repeat one email; the key lets the provider reject the duplicate. ## Two traps while testing the fix | Trap | What happens | |---|---| | `ack()` before a later message still throws | In production the acknowledged messages are safe, but every message after the throw still goes round again | | Testing under `wrangler dev` | The local queue simulation retries the whole batch when the handler throws or calls `retryAll()`, ignoring earlier `ack()` calls, and uses fewer default retries when `max_retries` is unset | The second is the subtle one. If you "fix" the bug by acking before the throw and then try it locally, the local simulation still redelivers the acknowledged messages, which makes a correct partial fix look broken and can send you chasing the wrong cause. Test the loop where nothing throws; that version behaves the same locally and in production. ## Cleaning up Once the consumer is fixed, the messages already on the dead-letter queue still need a decision: the good emails among them were sent, some of them four times, and must not be replayed blindly. A dead-letter consumer that re-sends only messages the provider has no record of, keyed by the same idempotency key, is the safe replay.
- Why might the fixed consumer still look broken when you test it under wrangler dev?The local queue simulation redelivers the whole batch whenever the handler throws or calls `retryAll()`, ignoring earlier `ack()` calls, and when `max_retries` is unset it uses fewer retries than production's default of 3. A loop where nothing throws behaves the same locally and when deployed, so test that version.
- Would setting max_batch_size to 1 fix the problem?It isolates the bad message, since each batch then holds one message, but it multiplies invocations and gives up batching for every good message. Per-message `ack()` and `retry()` get the same isolation and keep the batch.
- Is it safe to replay the dead-lettered batch once the consumer is fixed?Not blindly: the good messages in it were already sent, some of them four times. Replay through a dead-letter consumer that skips anything the provider has already accepted, keyed by the idempotency key generated when each message was produced.
- Only the malformed message is retried; the rest of the batch was acknowledged.
- Raising max_retries will eventually let the malformed message through.
- Calling batch.ackAll() first protects the batch from the poison message.
- The four sends prove the provider is retrying, not the queue.
- Acknowledging each message before the throw fixes the bug completely.