Forty callers a second read one shared entry and rewrite it under a version token, and adding callers lowers the rate of accepted writes - why, and what would you change?
answer
- cheap when collisions are rare
- one winner per generation of the entry
- discarded round trips scale with callers
- refusals per accepted write, per entry
- stop racing: server-side edit, or a claim
basics
~20 sOnly one caller per generation of the entry wins; every other in-flight attempt holds a token that the accepted write invalidates, so its work is discarded and repeated. More callers, more collisions, fewer accepted writes.
solid answer
~50 sOptimistic control is priced backwards from a lock. When collisions are rare it is nearly free - one extra field on the read, one comparison at the write, nobody blocked. When one entry is contended the same mechanism becomes pure waste: every attempt but one is discarded work, and each discarded attempt still cost a read round trip, a computation and a write. Adding callers raises the collision probability, so the store's operation rate climbs while accepted writes per second flatten and then fall. Two fixes stop the racing rather than pacing it: move the mutation to the server with a **server-side in-place update**, where the server can interpret the value, so the read-modify-write never crosses the network; or hold an **expiring claim** so callers wait their turn instead of colliding, trading discarded work for waiting. Bounding and spacing attempts is a mitigation, not a remedy.
go deeper
Get the shape of the cost: when two callers rarely want the same entry, the check costs almost nothing; when they all want one entry, all but one of them did their work for nothing and must do it again.
Explain the mechanism behind the inversion - one accepted write invalidates every token held at that moment - and be able to say which number reveals it, refusals per accepted write on that specific entry.
Show the judgment: name the three remedies, say which one the store in front of you actually offers, and say how each degrades. Bounded, spaced attempts belong in the answer as a mitigation, explicitly not as the fix.
Frame it as waste against waiting, and decide which the workload can carry. Then make the degradation observable, because this failure arrives as latency and load rather than as errors and will otherwise be diagnosed as a capacity problem.
## The bet, and the case it is designed for When collisions are rare, declare-and-recheck is close to free. The read carries one extra field, the write carries one comparison, nothing is held, and no caller is ever made to wait for another. That is the case optimistic concurrency control exists for: many callers, rarely the same entry at the same instant. In that regime it strictly dominates a claim, because a claim would make callers queue for a collision that was not going to happen. ## Why the curve inverts Now put many callers on one entry. At any instant several of them hold a token for the same generation of that entry. The first write accepted changes the entry, and that single write invalidates **every other token held at that moment**. So each accepted write is accompanied by however many attempts were in flight, each of which has already spent a read, a computation and a write round trip, and must spend them all again. Three consequences arrive together: 1. **Useful throughput flattens, then falls.** Accepted writes per second stop tracking offered load, and past a point they decline - the refused attempts compete with the accepted ones for the same server time, the same connections and the same entry. 2. **The store looks busier, not sicker.** Operations per second climb the whole way up. A dashboard watching operation rate sees growth; the waste is visible only as refusals measured against accepted writes. 3. **The losses are not evenly shared.** A caller that does more work between its read and its write is exposed for longer and loses more often, and nothing in the mechanism gives priority to a caller that has already lost. A slow caller on a hot entry can keep losing for as long as the load lasts. ## Telling which regime you are in - **Refused attempts per accepted write, per entry.** This is the waste multiplier and the only number that answers the question. A raw refusal count grows with traffic and tells you nothing. - **The distribution of attempts per success**, not its mean - a healthy mean hides the callers that take eight attempts every time. - **Latency of the whole retry sequence**, not of one conditional write. A single write stays fast under contention; the cost shows up in how many of them a caller needs. - **Which entry.** Contention is per entry, so an aggregate across the keyspace will always look fine. ## The remedies, and how each degrades | remedy | what it does | how it degrades, and when it is unavailable | |---|---|---| | server-side in-place update | the arithmetic or the edit happens at the store, so read-modify-write never crosses the network and there is no gap to lose | unavailable where the server cannot interpret the value; limited to edits the store can express | | expiring claim | callers take turns on the entry instead of racing, converting discarded work into waiting | the deadline must be sized against the protected work, or two callers hold it at once; queueing shows up as latency | | submitted program | one short program reads, decides and writes at the store as a unit, closing the gap without a token | a long-running program stalls the tier - head-of-line blocking where the server runs one operation to completion, and a held entry where worker threads lock the entry | | bounded, spaced attempts | stops a losing caller spinning and puts a ceiling on its latency | cheapest change and no fix: the entry is exactly as contended afterwards | ## What does not help - **More attempts.** Spending more capacity on the same race raises the operation rate and the latency, not the accepted-write rate. - **More callers or more capacity.** The bottleneck is one entry on one node; adding callers is the thing making it worse. - **Reading more often to be fresher.** A fresher read narrows the window without removing it. ## The decision in one line Optimistic control pays in **discarded work**; a claim pays in **waiting**; a server-side update pays by **requiring the server to understand the value** and by confining the decision to what the store can express. Choose by what the workload can tolerate and by which of the three the store you actually have provides - stores in this class differ on all three, and a design that assumes any one of them is universal will not survive a move to another store.
- Does the store suffer, or only the callers?Both, in different ways. The callers pay in latency and discarded work. The store pays real server time and connection capacity for every refused write, so a hot entry under optimistic retry raises the whole tier's load and can affect callers that never touch that entry - which is why the symptom often arrives as unrelated latency rather than as errors on the contended path.
- Why not just increase the retry limit until writes get through?Because the limit is not what is stopping them. Each extra attempt adds a full round trip's latency to the caller and more load to the entry that is already the bottleneck, which raises the collision probability for everyone else. A higher limit converts a visible failure into invisible latency, and delays the point at which anyone notices the design has stopped working.
- Is holding a claim strictly better once the entry is hot?It is usually better, but not strictly: the work is serialised either way, so throughput is bounded by one caller at a time in both. A claim replaces discarded work with orderly waiting, which is easier to reason about and cheaper for the store. It brings its own obligations - a deadline sized against the work, and a safe release - which is a different subject with its own failure modes.
saying these in an interview costs you the question
- Says optimistic retry always beats holding a lock.
- Reads a rising operation rate as healthy throughput.
- Measures refusals without dividing by accepted writes.
- Thinks a refused write is free for the server.
- Raises the retry limit and calls the problem fixed.
- Assumes a losing caller must eventually win.