How does deduplication work on an SQS FIFO queue, and why does it not give you end-to-end exactly-once processing?

## Two ways to supply the id A FIFO queue refuses a send that has no deduplication id, so you must pick one of two mechanisms. **Explicit id.** Pass `MessageDeduplicationId` on each `SendMessage` or `SendMessageBatch` entry. This is the better option in almost every case, because you control what "the same message" means. Use a stable identifier from your domain — the event id, the outbox row id, an idempotency key from the inbound HTTP request. Crucially it must be stable across the retry: if you generate a fresh UUID per attempt, the retry looks like a new message and dedup does nothing. **Content-based.** Set the queue attribute `ContentBasedDeduplication` to `true` and omit the parameter, and SQS computes a SHA-256 hash of the message **body**. Note that the hash covers the body only, not message attributes. That has two consequences people trip over. First, two genuinely distinct events with byte-identical bodies — "heartbeat", "retry job 5" — are treated as duplicates and the second is silently dropped. Second, adding a timestamp or a random field to the body defeats the hash entirely, so a retry that re-serialises with a new timestamp is no longer recognised. You can also override a content-based queue per message by passing `MessageDeduplicationId` explicitly. ## The window Deduplication lasts five minutes from acceptance, and that interval is fixed — there is no attribute to lengthen or shorten it. Inside it, a repeat send returns a successful response with the original message id and nothing is enqueued. After it, the same id is accepted as a new message. That scoping is deliberate. SQS is guarding against a specific event: a producer that sent successfully but lost the response, or an SDK that retried a socket timeout. Those retries happen within seconds. It is not a general-purpose idempotency store, and you should not lean on it to suppress a duplicate that your system might emit an hour later — that needs a durable key in your own datastore. In the default configuration the id is compared across the whole queue. High throughput mode narrows the comparison to the message group, which means the same id in two different groups is accepted twice. ## Why this is not exactly-once processing AWS documents FIFO queues as offering exactly-once *processing*, and interviewers like to push on the wording. What SQS guarantees is on the **send** path: a duplicate submission does not become a second queued message. The consumer path is unchanged. Walk the failure. A consumer receives message M, charges a card, writes a row, and the process is killed before it calls `DeleteMessage`. The visibility timeout expires, M becomes visible again, and some consumer receives it. Dedup is irrelevant here — nobody sent anything twice; the same single message was delivered twice. Any queue that guarantees at-least-once delivery must behave this way, because the alternative is losing the message when a consumer dies mid-work. The same applies if your handler succeeds but takes longer than the visibility timeout: the message is re-released while the first attempt is still running. ## What to do instead Treat FIFO dedup as one layer: - It removes duplicates caused by producer retries, which are common and annoying. - It does not remove duplicates caused by consumer failure, which are the ones that corrupt state. So the handler still has to be safe on a second run — typically by keying the effect on a business identifier and making a repeat converge to the same end state rather than applying a second time. That property is what actually delivers effective exactly-once semantics; the queue only reduces how often it is exercised. A useful summary line for an interview: FIFO deduplication is about the write to the queue, and idempotency is about the write to your database. They solve different halves of the problem, and only the second one is optional in the sense that you can skip it — at the cost of double-charging someone.