How does dead-letter queue routing actually get implemented differently in a broker like Amazon SQS or RabbitMQ versus a log-based system like Apache Kafka, where there is no built-in 'move failed message elsewhere' primitive?

## Broker-native routing: the broker does the work In broker-native systems like SQS and RabbitMQ, the DECISION that a message has failed enough times and the ACT of moving it are both broker responsibilities. - **SQS** attaches a **'redrive policy'** to a queue specifying `maxReceiveCount` and the ARN of a DLQ; SQS tracks `ApproximateReceiveCount` server-side per message, and crossing the threshold automatically moves it. - **RabbitMQ** instead uses a **dead-letter-exchange** (an `x-dead-letter-exchange` argument on a queue): a message is dead-lettered when it's explicitly nacked/rejected without requeue, when its TTL expires, or when the queue's max-length is exceeded — RabbitMQ then republishes it to the configured exchange, which is bound to a DLQ. In both cases consumer code doesn't have to build any queue-routing logic itself. ## Kafka has no server-side redrive Kafka is fundamentally different: topics are partitioned, ordered, immutable logs, and consumers don't 'pop' messages — they read at an offset and advance it independently. There's no broker-side concept of per-record state (delivered/failed/dead), so there's no server-side redrive. The application must: 1. Catch the exception around record processing. 2. Produce a copy of the failed record — often wrapped with headers like original topic/partition/offset, exception class, and stack trace — onto a dedicated DLQ topic via a producer. 3. Commit the consumer offset past that record so the main topic's consumption keeps advancing instead of reprocessing the poison record on every poll. Frameworks formalize this: - **Kafka Connect** exposes `errors.tolerance=all` plus `errors.deadletterqueue.topic.name` for sink connectors. - **Spring Kafka**'s `DeadLetterPublishingRecoverer` does the same for `@KafkaListener` consumers. - **Kafka Streams** exposes exception handlers for a similar purpose. ## The trade-off: simplicity versus flexibility The trade-off is simplicity versus flexibility. - **Broker-native DLQ** gives correct-by-default behavior with minimal code, but constrains you to what the broker's redrive policy supports — SQS's threshold, for example, is a single number, not conditional on error type. - **Kafka's app-level approach** is more code and more moving pieces — you must wire it into every consumer, keep offset-commit ordering correct, and avoid infinite loops where a DLQ-producer failure itself needs handling — but it's more flexible: you can inspect the exception type and route only permanent failures to the DLQ topic while relying on Kafka's normal offset-based redelivery (simply not committing the offset) for transient ones, or attach arbitrarily rich metadata as headers. ## The bugs this pattern produces 1. **Forgetting to commit the offset** past the failed record after successfully writing it to the DLQ topic is the most common production bug in Kafka's pattern, and it causes the same record to be reprocessed and re-published to the DLQ topic repeatedly on every consumer restart or rebalance, effectively duplicating it forever. 2. **The inverse bug** — committing the offset before successfully publishing to the DLQ topic — silently loses the record if the DLQ publish then fails. 3. In **RabbitMQ**, a common misunderstanding is what counts as a 'failed delivery': if a consumer crashes without explicitly nacking (e.g., its connection drops), the message is typically requeued rather than dead-lettered, so a crash-looping consumer might never trigger the DLX path the way an explicit reject-without-requeue would. ## A worked example A worked example: a Kafka Connect JDBC sink connector configured with `errors.tolerance=all` and `errors.deadletterqueue.topic.name=dlq-orders`. When a record can't be written to the database (e.g., a schema mismatch), the connector automatically writes the offending record, along with the exception message and stack trace as headers, to `dlq-orders`, and continues processing subsequent records from the source topic rather than stalling the whole connector task — mirroring, at the application/framework layer, exactly what SQS's redrive policy or RabbitMQ's DLX gives you natively.