Why must a deduplication record be written before the guarded work rather than after it, and what must the store offer?

## The interval that actually matters The mental picture behind writing the record last is that a duplicate arrives a comfortable while after the original succeeded. That is not when duplicates arrive. A sender retries because it did not learn the outcome, and the commonest reason it did not learn the outcome is that the first attempt was **still running** when its patience ran out. The duplicate therefore lands in the middle of the work, which is exactly the interval a record written at the end does not cover. Watch it fail. Suppose the guarded work is a charge that takes eight seconds and the sender gives up after five. | Step | Caller A | Caller B | Store | |---|---|---|---| | 1 | receives request `r`, starts the charge | | no record | | 2 | still charging | receives the same request `r` | no record | | 3 | still charging | looks for the record, finds nothing | no record | | 4 | charge succeeds, writes the record | starts the charge | record present | | 5 | | charge succeeds, writes the record | record present | Two charges, one record, and the record is perfectly correct and perfectly useless. No lifetime, no store and no amount of replication fixes this, because the code was never protected in the window that mattered. ## Claiming first The repair is to reverse the order: write the record **before** doing the work, and let the success or failure of that write decide whether this caller proceeds. The write has to be a **conditional create** - a write that succeeds only if the key is absent, completed in one operation - because the obvious alternative, reading the key and then writing it, is two operations with a gap between them, and two callers can both read nothing in that gap and both go on to write. As pseudocode - these verbs are invented and belong to no product: ``` key = deduplicationKey(requestIdentifier) claimed = createIfAbsent(key, state = "in-progress", lifetime = longestRetryWindow) if not claimed: return # an earlier arrival owns this work; do not perform it again performTheWork(requestIdentifier) markHandled(key, state = "finished", lifetime = longestRetryWindow) ``` The claim is the whole mechanism. The final write only records the outcome for anyone who later wants to know whether the work finished. ## What claiming first costs Claiming before the work introduces a failure the other ordering does not have. If the process dies between the claim and the completion, the store now holds a record saying this work is spoken for, and no retry will get past it - so the work never happens at all. You have traded a possible **twice** for a possible **zero**. That trade is usually the right one, because zero is recoverable and visible: the record's in-progress state has its own deadline, and once it passes, the next retry claims successfully and does the work. Choosing that deadline is a real decision: - **Too short**, and a slow but perfectly healthy attempt loses its claim while it is still working, so a retry starts a concurrent twin - the very thing the record exists to prevent. - **Too long**, and a genuine retry after a crash waits, doing nothing, until the deadline passes. A workable rule is to set the in-progress deadline a comfortable multiple of the slowest realistic completion time, and the finished deadline to the retry window from the sizing rule. They are different numbers doing different jobs. ## What the store must offer, and where stores differ The recipe is not portable by assumption. Before promising it, state what it needs: - **A conditional create in one operation.** Many stores in this class offer one. Some do not offer any way to make a write conditional on the key being absent, and on those this recipe simply cannot be built - the claim has to live where a uniqueness rule can be declared instead. - **A deadline attached by the same operation that creates the entry.** Where the store cannot do both at once, there is a moment in which the record exists with no deadline, and a crash in that moment leaves a permanent record blocking the work forever. - **The values can be opaque.** Nothing here needs the server to understand the value; a marker and a short state are enough, so this workload is buildable on stores that only hand back the bytes they were given. Two topology facts are worth saying before someone else says them. On a keyspace split across nodes, the claim is atomic on the node that holds it, which says nothing about the work on the other side of the call. And where a write is acknowledged before any copy has it, a failover can take the claim away while the work is running - after which a retry claims cleanly and does the work a second time. ## When ordering is not enough Correct ordering makes the record useful, not authoritative. If the guarded effect must not happen twice under any circumstances, the record is still an optimisation; the guard belongs in the durable store that records the effect and can declare the identifier unique.