A streaming ingestion consumer crashes on the same record after every restart — how do you isolate it?

## What a poison record actually is A poison record is a message whose processing fails **deterministically**. It is not slow, not unlucky, not affected by load. Every attempt produces the same exception at the same position: a payload that is not valid JSON, a field the schema says is an integer holding `"N/A"`, a value that violates a not-null constraint in the sink, an event whose enum member the code has never heard of. Because the consumer cannot process it and will not skip it, it also cannot commit progress past it. The partition or the queue stalls, lag climbs, and — if the runtime restarts the failed task — the system enters a tight crash-restart loop that also floods the logs. The diagnostic signature is the giveaway and interviewers expect you to name it: **the same offset or message id appears in the failure every single time**, and the error is identical. Compare that with a transient failure, where the failing position moves around, the error mentions the network or a downstream service, and a retry sometimes succeeds. ## Step one: classify, do not just count The most common bad implementation is "retry N times, then dead-letter, regardless of the error". It works beautifully on poison records and catastrophically during an outage. If the sink database is down, *every* record fails, every record exhausts its attempts, and the dead-letter store fills with the entire stream — correctly ordered data turned into an unordered backlog that someone must now replay by hand, often after the source's retention has expired. The dead-letter path must be reserved for errors that are **not going to succeed later**: - Deserialization and parse errors — the bytes are what they are. - Schema or contract violations — a required field is absent or the type is wrong. - Domain rejections — an amount outside its permitted range, an unknown reference key that will never resolve. And the halt-or-wait path is for errors whose success depends on something outside the record: connection failures, timeouts, authorization errors, capacity rejections. For those the right behaviour is to stop consuming and raise, not to divert — the pipeline should go down loudly rather than quietly convert an outage into permanent data displacement. A bounded attempt count still belongs in the design, but as a **safety net for misclassification**, not as the primary mechanism. ## Step two: divert with enough context to replay When a record is diverted, the dead-letter entry must carry the original payload untouched, the message key, the source position (topic, partition, offset, or queue message id), the timestamp, the consumer's version, and the exception class and message. Without the position you cannot prove where it came from; without the key you cannot reason about ordering; without the payload you cannot replay. Then commit past the record so the consumer resumes. ## Step three: make the divert loud Diverting is a decision to accept an incomplete target in exchange for a moving pipeline, and it must be visible. Emit a metric of dead-lettered records per minute, broken down by error class, and alert on two shapes: a **spike** (a schema change just made every record poisonous) and any **first appearance of a new error class**. A dead-letter destination that grows without anyone noticing is the streaming equivalent of silently dropping rows. ## The ordering problem nobody mentions until asked This is the part that separates a senior answer. In a keyed stream, records for the same key are ordered, and downstream state depends on applying them in order. Divert the record that sets `status = shipped` and the next record for that order — `status = delivered` — is applied on top of a target that never saw the shipped transition. That may be harmless if the sink is a last-write-wins upsert of full state, and actively wrong if the sink applies deltas, appends to a history, or drives a state machine. So the divert decision is per-sink-semantics, and the mitigations are concrete: for delta-style or state-machine sinks, record the diverted key so the consumer can skip or quarantine subsequent records for that key until the original is resolved; or accept the gap and, on replay, reconcile the key from the source rather than applying the stale message. A blind replay of an old dead-lettered record into a target that already holds newer state is how a replay makes things worse than the original failure. ## What good looks like Errors classified explicitly; deterministic failures diverted with full context; transient failures halting rather than diverting; attempts bounded as a backstop; a metric and alert on the divert rate and on new error classes; and a documented statement of what happens to per-key ordering when a divert occurs, with a matching replay procedure.