How long must a deduplication record live, and why is the first retry delay the wrong number to size it against?

## The rule, stated once A deduplication record must outlive the **retry window** - the longest span over which the same work can arrive again. Not the average gap between duplicates, not the request timeout, and not the sender's first backoff step. The last moment a duplicate can plausibly appear is the number the lifetime has to clear. The common failure is sizing against the wrong end of the distribution. Somebody sees that the client retries after two seconds, rounds up generously to five minutes, and ships it. Five minutes is an enormous multiple of the number they looked at, and it is still far too small, because the duplicate that actually costs money almost never comes from the client's own immediate retry. ## Where repeats actually come from | Source of the repeat | Who triggers it | Span it implies | |---|---|---| | The sender's own retry schedule | the client library, automatically | seconds to a few minutes, to its final attempt | | A consumer that stalled and later resumed | a stuck worker, a paused deployment | as long as the stall, often hours | | A scheduled reconciliation that resends anything with no recorded outcome | a nightly or hourly job | up to that job's interval, commonly a day | | A person: support replaying a stuck batch, a customer pressing submit again | a human | days, bounded only by policy | | A partner re-driving from their side after their own incident | someone else's runbook | whatever their runbook says | Deriving the number is mechanical once the sources are listed: 1. List every party that can send this work - including the ones inside your own company. 2. For each, write down the last moment it can send it again. 3. Take the maximum, not the median. 4. Add margin for clock differences between the parties and for the replay nobody documented. 5. Compare that number against what the tier will actually retain, and if the tier cannot promise it, decide deliberately where the real guard lives. Step five is the one candidates skip, and it is the interesting one. ## A configured lifetime is a ceiling, not a floor Attaching a deadline to the entry says when it stops being valid. It does not promise the entry will live that long. Two things shorten it without appearing in any code review: - **Removal under memory pressure.** A tier that is configured to make room by removing existing entries when it reaches its memory ceiling will remove some of yours. On a shared tier that is driven by other people's keys and other people's traffic peaks, so the retention of your records is coupled to workloads you do not own. Stores differ here in an important way: some refuse further writes at the ceiling instead of removing anything, and on those your records are safe but your writes start failing - a different failure that you would rather find out about in an interview answer than in production. - **A restart.** Some stores in this class keep nothing across one. Others reload a copy from disk that typically lags the most recent writes, which means the records for work currently in flight are the least likely to survive. So effective retention is at most the lifetime you configured, and your sizing is a request rather than a contract. ## The other direction: no deadline at all The mirror-image mistake is leaving the record permanent. These entries are minted one per request, so the key count grows with traffic forever, and on this tier the number of keys is usually what hurts before the bytes do. Permanent records also make removal under memory pressure likelier for every other workload sharing the tier, and they are among the hardest entries to clean up later, because from the outside you cannot tell which ones are still inside somebody's retry window. ## When the honest window is longer than you will pay for Suppose the honest retry window is thirty days, because a partner can re-drive a month of work after an incident, and you will not hold thirty days of per-request keys in memory. The answer is not to lie about the window. It is to split the job: - Put the durable guard where the effect is recorded - the store that writes the payout can declare the request identifier unique, so a second attempt fails when it tries to record the effect. - Keep a **short** volatile record sized to the common repeat, the sender's own retries, so the overwhelming majority of duplicates are recognised without touching the durable store at all. That arrangement is honest about what each layer is for: the volatile record is an optimisation with a stated hit rate, and the guarantee lives somewhere that cannot silently forget.