Why must the handler that applies events to a projection be idempotent, and what's a concrete technique to make an 'increment a counter' style update idempotent?

## What idempotency means here **Idempotent** means an operation produces the same end result no matter how many times it's applied — applying it once or five times leaves the system in the same state. For a projection handler, this matters because the realistic delivery guarantee from most event sources (Kafka, SQS, an event store's subscription, an outbox poller) is **'at-least-once,'** not 'exactly-once.' - A consumer can crash after processing an event but before durably recording that it did so, causing the same event to be redelivered on restart. - A transient network error can trigger an automatic retry of a message that actually already succeeded. - A consumer-group rebalance can briefly hand the same partition to two consumers. In every one of these cases, a projection handler can see the same event more than once, and if its update logic isn't idempotent, that redelivery corrupts the projection. ## The two general techniques There are two general techniques. 1. **Natural idempotency** is the first: design the update itself so that reapplying it changes nothing. 'Set this order's status to shipped' is naturally idempotent — running it twice leaves the status exactly the same as running it once. An upsert keyed by a stable entity ID ('insert this row, or replace it if it already exists with this ID') is likewise naturally idempotent. 2. **Explicit deduplication** is the second technique: track which events have already been applied — by event ID, or by a monotonically increasing offset/version — and check-and-skip before applying a new one. This works for operations that aren't naturally idempotent, most notably counters and other relative/delta updates ('increment by 1', 'append to a list'), where reapplying the same delta a second time necessarily changes the result. ## Why the broker cannot do it for you The reason idempotency has to be engineered in rather than assumed away is that true end-to-end exactly-once delivery is expensive and often unavailable. A message broker can offer exactly-once semantics within its own boundary — for example, transactional producers plus consumer-offset commits done atomically — but the moment a handler writes to an external store that isn't part of that same transaction (a Postgres table, a search index, a cache), you're back to at-least-once at the point that actually matters: a crash between writing the projection update and committing the consumer offset causes the event to be redelivered and reprocessed. Idempotency at the handler is what turns that unavoidable at-least-once delivery into **'effectively-once'** correctness without needing a distributed transaction spanning the broker and every downstream store. ## Complexity versus applicability The trade-off between the two techniques is complexity versus applicability. | Technique | The catch | |---|---| | **Naturally-idempotent updates** (absolute sets, upserts by key) need no extra bookkeeping and are simple to reason about | but not every operation is naturally expressible that way — aggregation, counting, and appending are inherently relative operations | | **The deduplication-ledger approach** is more general | but adds cost: extra storage for the dedup marker, a comparison on every event, and — critically — it only actually closes the gap if the dedup check and the projection mutation commit atomically as one write | If they're done as two separate steps against two separate stores or transactions, a crash between them reopens exactly the race idempotency was meant to close. ## Failure modes in production - In production, the most common failure is **a counter that silently drifts upward** from duplicate processing — nothing crashes, nothing errors, the numbers are just slightly wrong, and it's often only noticed much later when totals look 'off' relative to a source-of-truth reconciliation, at which point a full projection rebuild is usually the fix. - A related failure is **an unbounded dedup ledger** that grows forever because old entries are never pruned or windowed. - A third is **assuming events always arrive strictly in order** and treating a delta as if it always lands on top of the previous state — out-of-order redelivery can interact badly with a naive 'set to absolute value' pattern if the value carried in the event is itself a delta rather than a final state. ## The concrete pattern A concrete pattern: instead of `UPDATE counters SET count = count + 1 WHERE customer_id = X`, a projection stores `(customer_id, last_event_id, count)` and runs `UPDATE counters SET count = count + 1, last_event_id = :eventId WHERE customer_id = :cid AND last_event_id <> :eventId` — a single atomic conditional statement that increments only if this exact event hasn't already been applied, and is a safe no-op on redelivery. Webhook-consuming systems like Stripe's explicitly document this exact concern, warning integrators to expect duplicate deliveries and recommending they record an event's ID as processed atomically with applying its effect — the same discipline a projection handler needs internally.