A checkout pipeline spans three hops: an API gateway with client idempotency keys, a Kafka topic with an at-least-once consumer, and a downstream call to a third-party shipping-label API that has its own idempotency-key parameter. Each hop is individually idempotent. Why can the end-to-end pipeline still produce duplicate shipping labels, and how would you close that gap?

## Per-hop idempotency versus end-to-end idempotency A pipeline's overall correctness under retries is not the sum of each hop's individual idempotency - it's the sum of each hop's idempotency composed with how identity is threaded between hops. This is a subtle failure mode that only shows up under scale and multi-hop architectures, and it's a common source of production incidents in systems that individually 'did everything right' at each layer. ## The pipeline, hop by hop Consider the described pipeline. 1. The **API gateway** correctly deduplicates client-submitted checkout requests using a client-provided idempotency key - this protects against the client retrying the same checkout twice. It publishes an internal event ('checkout confirmed for order O') to Kafka. 2. The **Kafka consumer** is at-least-once, meaning it can process that event more than once (crash-and-redeliver, consumer-group rebalance replaying from the last committed offset) - individually fine, since the consumer is expected to handle redelivery. 3. On each processing attempt, the consumer calls a **third-party shipping-label API**, which itself accepts an idempotency-key parameter and promises not to issue two labels for the same key - also individually fine, by design, on the third party's side. ## Where the seam opens The gap appears at the seam between the Kafka consumer and the shipping-label call. If the consumer's code path generates the idempotency key it sends to the shipping API **freshly on each invocation** - for instance, generating a new random token right before the outbound call, or deriving the key from wall-clock time - then two different processing attempts of the same Kafka event (one that crashed after calling the API but before committing its offset, and the redelivered retry) will produce two different keys sent to the shipping API. From the shipping API's perspective, these look like two entirely unrelated, legitimate requests - its own idempotency mechanism has no way to know they both trace back to the same order, because idempotency keys only dedup calls that share the same key. The result: two shipping labels for one order, despite every individual component in the chain being correctly implemented and individually idempotent. ## The principle, and the fix The general principle this exposes is that idempotency needs a **stable identity that survives the entire causal chain** of retries, not just protection localized to each hop. The fix is to derive every downstream idempotency key deterministically from an identity that is fixed at the point the causal chain begins - typically the original order ID or event ID - rather than generating a fresh token at each hop. Concretely: the shipping-label call's idempotency key should be a **deterministic hash of the order ID plus a fixed operation name**, computed from data already present on the Kafka event, so that no matter how many times that event is redelivered and reprocessed, the exact same key is sent to the shipping API every time, and the shipping API's own dedup catches the duplicate reliably. ## Two patterns worth naming This connects to two well-known architectural patterns worth naming. 1. **First, the outbox pattern**: rather than 'consume event, then separately call the external API,' a robust design has the consumer write its intent to call the shipping API (including the deterministic key) into a local outbox table in the same transaction as any local state change, and a separate, idempotent publisher process reads the outbox and makes the actual external call, retrying safely because the outbox row itself is the single source of truth for 'did we already decide to make this call,' decoupling that decision from 'did the external call complete.' 2. **Second**, this is a scaled-up instance of the same claim-atomicity problem discussed for a single dedup store: at pipeline scale, the effective 'dedup store' is the union of every hop's dedup mechanism, and the engineering discipline required is making sure the key itself is propagated as a stable, derived value through the whole chain, not regenerated at any point. ## The same advice elsewhere A concrete real-world parallel: this exact class of bug is why - guidance for chained serverless pipelines commonly recommends deriving idempotency keys from the original event's message ID or a business-level natural key (order ID, payment intent ID) rather than generating a new key inside each function invocation; - documentation for idempotency-key-based payment APIs stresses that keys should be deterministically derivable from the operation's business context specifically to survive being replayed through intermediate systems the client doesn't control.