One entry takes thousands of read-modify-write attempts per second: how do a server-side edit, declare-and-retry, and an expiring claim each degrade?

## Why contention is the whole question At one attempt a second, all three remedies work and nothing distinguishes them. The interview question is what happens at ten thousand. Each remedy has a different currency: one spends server time, one spends wasted round trips, one spends waiting. Knowing which currency you are spending is the difference between a design that degrades and one that falls over. | remedy | what a losing caller does | throughput as contention rises | where it stops existing | |---|---|---|---| | server-side in-place update | nothing — there is no loser | flat, bounded by server time at the entry | where the server cannot interpret the value | | declare-and-retry | re-reads and attempts again | falls; most attempts produce nothing | nowhere, but the cost becomes unacceptable | | expiring claim | waits, then gives up | roughly one holder per hold time | where the work outlasts the deadline | ## Server-side in-place update The change is expressed as something the server can do itself, so nothing crosses the network between reading and writing and there is no gap to lose. Its behaviour under contention is the best of the three: - Every attempt succeeds; there is no retry path to tune and no failure mode to explain. - The cost appears as **queueing at the entry** — under one-operation-at-a-time execution every caller waits behind the operation in progress; under per-entry locking they contend for that entry's lock while other entries proceed. Either way the operations are short, so the queue drains. - Its limits are real: the server must interpret the value, and the change must be one the server knows how to make. A decision that depends on what was read — "increase this only if the other field is below a threshold" — is not expressible as an in-place update, and needs a short submitted program instead, which is a different leaf's subject. ## Declare-and-retry This is the remedy that people reach for first and that contention punishes hardest. The caller either reads a version token with the value and presents it on write, or names the entries it read before submitting a group that is abandoned if one of them changed. - Each lost attempt has already spent a round trip to read, work to compute, and a round trip to be refused. **Under heavy contention most of the traffic to that entry is attempts that will not land.** - The system is not deadlocked and never reports an error — callers are busy and making no progress, which looks like ordinary load. - The distribution is unfair: a caller with a slow compute step between its read and its write loses to faster callers repeatedly, so the tail latency is far worse than the average suggests. - Adding retries makes it worse, not better: more attempts against the same entry means more refusals. The practical rules that follow are: **bound the attempts**, decide explicitly what happens at the bound, and instrument attempts-per-success — that ratio is the health metric for this remedy, and it is the number that tells you when to switch. ## The expiring claim A claim converts a race into a queue. One caller creates a marker entry only if none exists, does the work, and releases it; everyone else waits. - Throughput is bounded by **one holder per hold time**, so if the protected cycle takes five milliseconds, the entry supports roughly two hundred updates a second no matter how many callers there are. - The cost is wasted waiting rather than wasted attempts. That is the right trade when the protected work is expensive — a large document to parse, a call to another system inside the cycle — because losing that work repeatedly is worse than queueing for it. - The deadline is the trap. It exists so that a caller that dies does not block the entry forever, which means a claim whose deadline is shorter than the work it guards expires while the holder is still working, a second caller takes it, and both proceed believing they are alone. That is the original race with more moving parts and a false sense of safety. - Waiting callers are consuming their own request timeouts, so a claim held under load pushes latency into the caller's callers. ## Reading the signal in production The three remedies fail in ways that look alike from the outside — requests get slower — and quite different in the numbers: 1. If **server time per operation stays flat** while caller-side latency climbs, the time is being spent in retries or in waiting, not in the store. 2. If **attempts-per-success** rises above one, declare-and-retry is the thing degrading. 3. If callers are idle-waiting and throughput sits at a ceiling that does not move when you add callers, the claim's hold time is the ceiling. ## When none of the three is acceptable Sometimes the entry should not be the coordination point at all. Ask whether the value must be exact, whether the update must be synchronous, and whether this tier — which offers no isolation level, no rollback and no cross-entry guarantee — should own the invariant. "None of these, and here is where the invariant moves" beats picking the least bad of three.