Once messages start landing in a dead-letter queue, what operational practices — alerting and a 'parking lot' replay strategy — turn the DLQ from a silent graveyard into something actionable, and what should a replay process actually check before resending a message?

## The parking lot pattern The **'parking lot' pattern** treats the DLQ as a review queue rather than something auto-retried or auto-discarded. A separate replay tool, job, or console reads from the DLQ, allows filtering and inspecting messages (by error type, age, or originating topic), and pushes selected ones back onto the original queue or topic, typically after either: - the code that caused the failure has been fixed and deployed, - the message itself has been edited or patched to correct bad data, - or a downstream outage that caused a burst of failures has resolved. Concretely, for SQS this is often a small tool that reads a batch from the DLQ and, for each message, either deletes it (discard) or sends it back to the source queue and deletes it from the DLQ once re-enqueued. ## Why draining and alerting matter This exists because, without active draining, a DLQ just accumulates — the business operations those messages represent (unshipped orders, unprocessed payments, unsent notifications) stay undone, and the team has no early signal that a change broke something until customers complain. Alerting on DLQ depth or growth rate converts an invisible failure into a page, which matters because DLQ failures are inherently asynchronous and easy to miss compared to a synchronous request returning an error directly to a user. ## The trade-off The trade-off is between alert noise and blind spots. - **Alerting**: paging on every single DLQ message creates alert fatigue if the system has ordinary background noise from occasionally-bad input. A better approach alerts on rate-of-growth or absolute depth crossing a threshold (e.g., 'DLQ depth grew by more than 20 in 5 minutes' or 'oldest message older than 1 hour') and routes lower-severity single-message events to a daily digest or ticket rather than a page. - **Replay** has a similar trade-off: replaying too eagerly (auto-replay on a timer) risks reprocessing messages whose failure cause hasn't actually been fixed, potentially recreating another retry storm; replaying too conservatively (fully manual, one at a time) doesn't scale when an outage dumps thousands of messages into the DLQ at once, so tooling needs both a bulk 'replay all matching this error signature' path and safeguards like idempotency keys. ## Failure modes Failure modes in production include: 1. **Silent DLQ growth with no alarm**, where a team discovers weeks later that a deploy broke deserialization for one message type and thousands of customer-impacting messages sat stuck the whole time. 2. **Replaying without checking idempotency**, where a message that had actually partially succeeded (payment charged, but the order-confirmation email step failed and it dead-lettered) gets replayed from the top and double-charges the customer. 3. **DLQ retention expiring before anyone drains it** — SQS DLQs default to 4 days retention unless raised, so a late or missing alert can mean the message ages out and is permanently deleted, i.e., real business data loss, not just delayed processing. ## A worked scenario A worked scenario: an e-commerce platform's order-confirmation-email consumer dead-letters when the email provider's API returns validation errors for malformed addresses. A CloudWatch alarm on the DLQ's `ApproximateNumberOfMessagesVisible` fires when depth exceeds 50, opening a ticket for the on-call engineer (not a wake-up page, since this consumer is non-critical). The team runs a scheduled weekly 'parking lot' review where they batch-inspect DLQ messages by error signature, discard ones caused by genuinely bad customer input (typo'd emails — nothing to fix), and bulk-replay the ones caused by a since-fixed provider outage, using the message's original idempotency key so already-sent emails aren't sent twice.