After a supervisor decides to remediate a timed-out step by retrying it, why must the underlying operation the agent performs typically be idempotent, and what happens if it isn't?

## Why idempotency is structural here, not decorative Idempotency is not a nice-to-have decoration on this pattern, it's a structural requirement created directly by the ambiguity of timeout-based failure detection. When the supervisor sees a step's deadline pass with no terminal status, that single observable fact - 'no acknowledgment arrived in time' - is consistent with several very different underlying realities: - the remote call never reached the service at all (network partition before the request); - the service received it but crashed before responding; - the service processed it successfully but the response was lost on the way back; - or the service is simply still working on a slow request. The supervisor has no way to tell these apart from where it sits, and building a perfectly reliable distributed detector that could tell them apart is fundamentally hard (this is essentially the same uncertainty behind the **two generals' problem**). So the pattern's designers make a pragmatic choice: treat a timeout as 'unknown, possibly failed' and default to retrying, but push the burden of safety onto the operation itself rather than onto the detection mechanism. ## How it is actually implemented 1. **The key is created once.** Concretely, idempotency in this context is usually implemented via a client-generated idempotency key or request ID that's created once, when the scheduler first dispatches the step, and reused on every retry of that step. 2. **The service checks that key before acting.** The remote service, on receiving a request, checks whether it has already processed that key; if so, it returns the previously computed result without re-executing the side effect (charging money, decrementing inventory, sending a message) a second time. This shifts deduplication to the side that can actually know for certain whether the effect already happened - the service holding the state - rather than the caller, which can never be certain from a timeout alone. Payment processors are the canonical example: Stripe's API, for instance, accepts an idempotency key header specifically so that callers can safely retry a charge request after an ambiguous timeout without risking a double charge. ## What it costs The trade-off is that idempotency isn't free: - it requires the remote service to persist a record of processed request IDs (often with a retention window, since keeping them forever is its own storage cost); - and it requires every write path that a retry could hit to check that record before acting, adding a lookup to the hot path of every request. For read-only or naturally idempotent operations (setting a value to a fixed state, as opposed to incrementing a counter) this cost is often near zero, but for operations with genuine external side effects - sending an email, calling a third-party API you don't control, moving physical inventory - you either need the downstream system to support idempotency keys itself, or you need to add your own deduplication layer in front of it, which is extra engineering effort that some teams skip under time pressure. ## The failure mode when a team skips it The failure mode when a team skips this is exactly the 'invisible until it isn't' kind that's expensive in production: everything looks fine in testing because timeouts rarely fire in a controlled environment, then under real network conditions - a brief partition, a slow downstream dependency, a garbage-collection pause on the remote service - a step legitimately times out after actually succeeding, the supervisor retries it per policy, and the operation runs twice: - For a payment this shows up as a customer service ticket and a chargeback. - For an inventory reservation it shows up as overselling. - For a notification it shows up as an annoyed user getting the same email twice. These bugs are notoriously hard to reproduce because they require the exact timing window where the original call succeeded just after the deadline check ran, so they tend to surface only at production scale and get initially misdiagnosed as 'random' data corruption rather than traced back to the retry policy. ## Where you have already seen it A well-known real-world illustration is how orchestration engines like Temporal handle this: each activity invocation carries a deterministic identity tied to the workflow's execution history, and activities are documented as requiring idempotency specifically because the engine's own retry policy, driven by the same timeout ambiguity described here, may invoke the same activity code more than once for a single logical step. The pattern's designers don't try to solve the detection ambiguity - they just document that the burden shifts to the activity author to make the operation safe to repeat.