A single malformed message lands in position 500 of a Kafka partition that a consumer group is processing sequentially, and the consumer's deserializer throws every time it hits that offset. How does the resulting outage differ from what would happen if the same bad message had instead been delivered via an SQS queue, and why?

## Why one bad offset stops a partition Kafka partitions are strictly ordered, and a consumer's poll-and-process loop processes offsets in that order within a partition - a deliberate, valuable feature for causally related events. If the consumer throws an exception processing offset 500 and simply retries the same poll, it will throw again on 500 forever without advancing, and every subsequent offset in that partition, even perfectly fine unrelated messages, never gets processed. Throughput for that partition effectively drops to zero, either silently or noisily depending on whether the crash triggers a restart loop. ## Why a queue shrugs it off SQS, by contrast, treats each message as independent. A message is delivered, retried according to its visibility timeout and `maxReceiveCount` setting, and either succeeds or, after enough failures, is redirected to a configured dead-letter queue - critically, per message. Message #500 failing has zero effect on messages #501 through #1000, which continue to be received and processed by the same or other workers in parallel. ## The difference is structural The difference is structural, tracing back to the same root distinction covered elsewhere in this topic. - **Kafka's value proposition** is a durable, strictly ordered, replayable sequence, useful for causally dependent events - for instance, an 'account created' event must be processed before an 'account updated' event for the same account if both share a partition key. Strict ordering is inseparable from the possibility of one bad element blocking everything after it in that same ordered unit. - **SQS** deliberately gives up ordering (or scopes it narrowly, per FIFO message-group) in exchange for per-message independence and resilience to individual poison messages. ## The trade-off both ways The trade-off is real on both sides. - **Accepting Kafka's ordering** buys correctness for causally dependent processing at the cost of a systemic fragility: any one malformed or logic-triggering message becomes a single point of stall for everything downstream of it on that partition, until a human intervenes. - **Accepting SQS's independence** buys fault isolation for free - one bad apple doesn't spoil the barrel - at the cost of having no ordering guarantee across the whole queue, unacceptable for workloads that truly need strict sequencing. ## What it looks like, and what to do In production, a Kafka poison-pill stall typically shows up as consumer-group lag climbing linearly for one specific partition while others stay flat, often paired with a crash-loop-backoff pattern in orchestration logs. The mitigations: 1. **The standard mitigation** is to wrap message processing in error handling that, after N retries, explicitly commits past the bad offset and routes the raw bytes to a separate 'Kafka DLQ' topic built by the application itself, since Kafka provides no such mechanism natively at the raw consumer-API level - this restores throughput for the rest of the partition at the cost of consciously skipping data, which must be logged and alerted so someone investigates it. 2. **Choosing a higher-cardinality partition key** (e.g., per-customer rather than one giant shared key) also limits blast radius, since a stall only affects entities hashed to that one partition, not the whole topic. ## A worked scenario A worked scenario: an inventory-sync service consumes `InventoryAdjusted` events keyed by warehouse ID. A producer bug emits one event with corrupted JSON for warehouse `wh-042`. - **The stall.** The consumer's partition serving `wh-042` stalls completely - every subsequent adjustment for that warehouse, and any other warehouse whose key happens to hash to the same partition, stops being applied, silently drifting inventory counts, while warehouses on other partitions continue normally. - **On-call is paged by a lag alert an hour later**, diagnoses the poison message via offset inspection, and builds a one-off tool to skip past offset 500 for that partition and route it to a DLQ, only after which inventory sync resumes for the affected warehouses. - **Contrast** this with an SQS-based design, where that same corrupted message would have automatically landed in a dead-letter queue after 3 failed attempts within seconds, with zero impact on any other warehouse's messages.