In distributed systems, what is the difference between at-most-once, at-least-once, and exactly-once message delivery, and which of these can actually be guaranteed over an unreliable network?

## What a delivery guarantee promises Delivery semantics describe the guarantee a messaging system makes about how many times a message sent by a producer will be observed by a consumer, given that networks, processes, and machines can fail at any point. There are three canonical categories, and understanding why only two of them are actually achievable is foundational to distributed systems design. ## At-most-once: simplicity paid for with silent loss **At-most-once delivery** means the sender transmits a message a single time and does not retry, regardless of whether it receives confirmation. If the message or its acknowledgment is lost in transit, the message is simply gone. This is the semantics you get 'for free' from a bare fire-and-forget call, or from a system that explicitly disables retries. - **Its virtue is simplicity and low latency:** no retry logic, no deduplication bookkeeping, no risk of duplicate side effects. - **Its cost is silent data loss** - a metrics sample or a log line can vanish and nobody notices. - **It is appropriate only when occasional loss is tolerable** (best-effort telemetry, heartbeats) and never when correctness depends on every event landing. ## At-least-once: the practical default **At-least-once delivery** means the sender keeps retrying until it receives confirmation that the message was received. It retries on: - a timeout - a missing acknowledgment - an explicit failure signal This guarantees no message is silently dropped, but it introduces the possibility of duplicates: if the message actually arrived and was processed, but the acknowledgment got lost on the way back (or the consumer crashed after processing but before acking), the sender has no way to tell the difference between 'nothing happened' and 'everything happened, only the ack got lost.' From the sender's point of view, both look identical - a timeout. So it retries, and the receiver may see the same logical message twice. Almost every practical durable messaging system (default **Kafka** producers, standard **SQS** queues, **HTTP** retry-on-timeout logic composed with application-level acking) is built on at-least-once semantics, because guaranteeing 'not lost' is achievable and cheap, while eliminating duplicates at the transport layer is not. ## Why exactly-once delivery is out of reach **Exactly-once delivery** - a message observed by the consumer precisely one time, no more, no less - is the semantics everyone wants and, as a pure delivery guarantee, cannot be achieved over an unreliable network. The reasoning traces to the **Two Generals Problem**: two parties trying to reach agreement over a channel that can lose messages can never be fully certain the other side received the last message, because the acknowledgment of that message could itself be lost, requiring an acknowledgment of the acknowledgment, ad infinitum. Concretely: the sender transmits, then waits. If nothing comes back, it cannot distinguish three cases: 1. the request was lost before reaching the receiver; 2. the request arrived and was processed but the response was lost on the way back; 3. the request arrived and is still being processed slowly. The first case calls for a safe retry; the second calls for not reprocessing; the sender cannot tell them apart from a timeout alone, and no finite number of extra round trips removes this ambiguity. ## What production systems actually sell you What production systems market as 'exactly-once processing' is not exactly-once delivery at the network layer at all: - **Kafka's** transactional plus idempotent producer with `read-committed` consumers - **Flink's** exactly-once checkpointing - **Stripe's** idempotency-key API It is at-least-once delivery at the transport layer, combined with idempotent processing at the application or storage layer, so that even though the same message may be delivered and handled multiple times, its side effects are applied only once. This composite guarantee is usually called **'effectively-once'**: the wire can duplicate freely, but the receiver recognizes a duplicate and discards or no-ops on the repeat. This is the crucial mental shift for engineers: exactly-once is not a delivery property you can buy from your message broker - it's an engineering outcome you build by pairing at-least-once delivery with idempotency on the consumer side. ## The concrete case, and where it leaks A concrete example: **Kafka's exactly-once semantics (EOS)** for stream processing combines 1. an idempotent producer - each message carries a producer ID and sequence number so the broker can drop duplicate retries at write time; 2. transactional writes across `input-offset-commit` and `output-produce`. Even here, the guarantee is scoped to within the Kafka cluster - the moment that pipeline writes to an external system without its own idempotency mechanism, the 'exactly-once' property leaks and duplicates can reappear, which is why idempotency keys and dedup stores matter even in ecosystems that advertise exactly-once out of the box.