In a Pipes and Filters pipeline built on message queues, a filter reads a message, starts processing it, and crashes before acknowledging completion to the queue. What failure modes does this create for the pipeline, and how do you design filters to survive it?

## Two steps that can be interrupted This scenario is the central reliability challenge of building Pipes and Filters on real message queues, and it stems from the fact that 'processing a message' and 'acknowledging a message' are two separate steps that can be interrupted between each other. Almost all cloud message queues and brokers (Amazon SQS, Azure Service Bus, RabbitMQ, Kafka consumer groups) default to **at-least-once delivery semantics**: a consumer pulls a message, and the broker only removes it (or advances the consumer's offset) once the consumer explicitly acknowledges completion. If the filter crashes, gets killed, or times out after pulling the message but before acknowledging, the broker has no way to know whether work actually completed, so its only safe option is to make the message visible again for another attempt — either by the same filter instance after restart or by a different instance in the same consumer group. **The result:** the same message gets processed more than once, and any side effect that filter performs can happen more than once too: - writing to a database - calling a downstream API - publishing a message to the next pipe ## Designing every filter to be idempotent The standard, non-negotiable fix is designing every filter to be **idempotent**: processing the same message N times must produce the same observable end state as processing it exactly once. There are several concrete techniques. 1. The cleanest is making the underlying operation naturally idempotent — an upsert (insert-or-replace by message ID) instead of an insert, a conditional write ('set balance to X only if version = Y') instead of a blind increment, or a PUT-style 'set the final state' operation instead of a PATCH-style delta. 2. Where the operation isn't naturally idempotent (e.g., 'charge a credit card' or 'send an email' — calling twice really does have two effects), the filter needs an explicit deduplication layer: extract a unique identifier from the message (or generate one deterministically, such as a hash of its content), check a durable store (a database table, a cache with TTL) for whether that ID was already processed before doing the real work, and record it as processed atomically with the side effect. 3. Many managed queues also offer built-in deduplication windows (e.g., SQS FIFO queue content-based dedup) that catch exact-duplicate redeliveries within a time window, but that's a narrower guarantee than application-level idempotency and doesn't cover all failure timing. ## The poison message A second, distinct failure mode from the same scenario is the **poison message**: a message that, for some reason (malformed data, a bug triggered only by this specific payload, a downstream dependency that always rejects it), causes the filter to crash or fail every single time it's attempted. - Without a limit, at-least-once redelivery means this message gets retried forever, consuming worker capacity and — worse — typically blocking ordered processing behind it if the pipe preserves order (as with a single partition in Kafka or a FIFO queue), stalling every message that arrived after it. - The mitigation is a bounded retry count per message plus a **dead-letter queue (DLQ)**: after N failed attempts, the broker (or the filter's own retry wrapper) moves the message off the main pipe into a separate DLQ where it stops blocking the pipeline, and an operator or an automated remediation process can inspect and reprocess it later. - Nearly every managed queue (SQS redrive policies, Service Bus dead-lettering, Kafka via a manual DLQ topic pattern) supports this natively. ## Partial side effects A third, more subtle failure is **partial side effects**: if a filter's job is 'write to database, then publish to the next pipe' and it crashes between those two steps, a naive retry either re-does the DB write (fine if idempotent) or skips straight to publishing without the DB write ever having happened reliably, depending on how the crash-and-retry logic is structured. This is where patterns like **transactional outbox** (write the DB change and the 'to-publish' record in one local transaction, then a separate relay process publishes from the outbox reliably) become relevant for filters that need to guarantee both a state change and a downstream message happen together, or neither does. ## Pushing the guarantee into the system that matters A concrete real-world instance: an order-fulfillment pipeline's `charge-payment` filter reads an order-created message, calls a payment gateway, and publishes a payment-completed message. If it crashes after the gateway call succeeds but before publishing, at-least-once redelivery causes it to run again — and without idempotency, the customer would be charged twice. Real systems solve this by passing an idempotency key (often the order ID or message ID) to the payment gateway itself, since most major payment processors (Stripe, for instance) support idempotency keys precisely so that a retried charge request with the same key returns the original result instead of creating a second charge — pushing the idempotency guarantee down into the external system that actually matters, rather than trying to solve it only at the filter's own boundary.