In a Redis cache-aside setup, a slow read that missed can write its already-outdated value into the cache after a concurrent update has invalidated that key. How do you keep the stale value out, and where does SET with the NX flag help?
answer
- reader reads old row, writer deletes, reader writes back
- window = DB read → cache write
- short TTL bounds the damage
- SET ... NX: fill only, never clobber
- versioned CAS via Lua or WATCH/MULTI = real fix
basics
~20 sThe reader loaded the old row before the write committed, then wrote it back after the invalidation. Mitigations: keep TTLs short so damage is bounded; delete the key again after a short delay; write back with SET NX so a reader cannot clobber an entry a writer has already refreshed; or make the write-back conditional on a version stored with the value.
solid answer
~1 minThe race: reader R does `GET` (miss), starts a slow database read and sees version 1. Writer W commits version 2 and deletes the key. R then writes version 1 back with a fresh full TTL. The cache is now stale for the entire TTL, and the invalidation that should have fixed it already happened. Mitigations, cheapest first: - **Short TTL.** Bounds the damage without any extra machinery. Often sufficient. - **Delayed second delete.** After the write commits and the first delete, schedule another delete a few hundred milliseconds later, past the window in which in-flight readers can still write back. - **`SET key value EX ttl NX`** on the repopulation path. NX means "only if the key does not exist", so a reader can only *fill an empty slot*; it can never overwrite a value that a writer has since placed there. It does **not** fix the case where the writer only deleted, since the slot is genuinely empty then — but it does remove reader-clobbers-writer and reader-clobbers-fresher-reader. - **Versioned write-back.** Store the row version alongside the value and make the write-back a compare-and-set — a small Lua script or a `WATCH`/`MULTI` sequence that refuses to install a version older than what is present. I pick by cost of staleness: short TTL plus write-then-invalidate for most caches, delayed double delete or versioned CAS where a wrong value is expensive.
code
text · 10 lines# reader (slow) -- writer (fast)
GET product:v1:915 -> (nil)
# UPDATE ... COMMIT
# UNLINK product:v1:915
SET product:v1:915 "price=10" EX 600 NX
OK # NX still succeeds: the slot was genuinely empty
# but NX does prevent this one:
# writer repopulates: SET product:v1:915 "price=20" EX 600
# slow reader: SET product:v1:915 "price=10" EX 600 NX -> (nil), rejectedgo deeper
Be able to describe the interleaving in order and say that a short TTL limits how long a stale value can survive.
Explain write-then-invalidate ordering, what SET ... NX does and does not prevent, and the delayed second delete.
Compare the mitigations by cost and residual risk, implement versioned write-back with a Lua script or WATCH/MULTI, and instrument cache-versus-database disagreement.
Frame it as a staleness budget per data class: which fields may ever be served from a cache, what invariants a versioned write-back needs, and when the answer is to not cache the field at all.
## The exact interleaving Write the timeline out; the bug is only visible in the ordering. ``` T1 Reader GET product:v1:915 -> nil (miss) T2 Reader SELECT ... WHERE id=915 -> price=10 (slow: GC pause, slow query, network) T3 Writer UPDATE ... price=20; COMMIT T4 Writer UNLINK product:v1:915 -> key absent (it was already absent) T5 Reader SET product:v1:915 price=10 EX 600 ``` After T5, Redis holds price=10 with a full 600-second TTL, and the database holds 20. No further event will correct it. This is *not* fixed by choosing write-then-invalidate ordering — write-then-invalidate is still the right order, and this residual race is what remains after you have chosen it. The window is the gap between the reader's database read completing and its cache write landing (T2→T5). Usually microseconds to a few milliseconds; occasionally a full GC pause or a retried connection makes it hundreds of milliseconds. Because it needs an unlucky interleaving with a write to the same key, it is rare — which is exactly why it survives testing and shows up as an unreproducible "the page showed the old price" ticket. ## Mitigation 1: bounded staleness (short TTL) The cheapest correct answer. If entries live 30-60 seconds rather than an hour, a lost race self-heals quickly. Many caches genuinely need nothing more, and saying so in an interview is a sign of judgment rather than of ignorance — provided you can also describe the stronger options and when they are warranted. ## Mitigation 2: delayed double delete After committing and deleting, schedule a **second** delete some hundreds of milliseconds later (a delay queue, a timer, a small worker). Any in-flight reader that was going to write a stale value has done so by then, and the second delete removes it. The next reader loads fresh. Properties: simple, no coordination, works regardless of value shape. Costs: you need somewhere reliable to run the delayed action (a crashed process silently drops it), it produces one extra miss, and the delay is a guess — too short and you miss slow readers, too long and you throw away useful entries. It is a probability reducer, not a proof. ## Mitigation 3: SET ... NX on repopulation `SET key value EX ttl NX` writes only if the key does **not** exist. Applied to the cache-aside fill path, it changes the reader's role from "author" to "filler of an empty slot". What that buys: - A reader can never overwrite a value a **writer** put there (if your write path populates rather than only deletes). - A slow reader can never overwrite a **fresher reader's** fill. Between two concurrent readers, the first to arrive wins and the late, older one is rejected — which is exactly the direction you want. What it does **not** buy: in the delete-only invalidation flow above, the key is genuinely absent at T5, so NX succeeds and the stale value still lands. NX is a necessary but insufficient guard; it removes clobbering, not the empty-slot fill. It composes well with the other techniques, and it costs nothing — the same single command with one more flag. Note that `SET` returns nil when NX rejects the write, so the client can observe and count rejections. Be careful not to confuse this use of NX with the *lock* use of NX for stampede control; here NX is guarding the value, not serializing the loaders. ## Mitigation 4: versioned compare-and-set write-back The robust fix: carry a monotonic version with the value — a row version column, an updated-at timestamp with sufficient resolution, or a transaction id. Store it inside the payload or in a companion field, and make the write-back conditional: install the new value only if its version is greater than the one currently cached (or if nothing is cached). Redis has no built-in conditional-on-content SET, so you implement it as either a tiny Lua script (read current, compare versions, set or skip — atomic because scripts run to completion on the server) or a `WATCH key` / `MULTI` / `EXEC` optimistic sequence where EXEC aborts if the key changed. A hash with a `ver` field plus a script is the common shape. This converts the write-back from last-writer-wins to highest-version-wins, which closes the race properly. The costs are real: a script or transaction on every fill, a version source you must trust to be monotonic (clock-derived timestamps across machines are not), and more code in a hot path. ## Choosing Ask what a stale value costs for the length of a TTL. A product description: short TTL is fine. A price, an entitlement, a feature flag, a balance: use versioned write-back, or do not cache the field at all and read it from the database. A middle tier — user profile, catalog metadata — is well served by write-then-invalidate + short TTL + NX fills + a delayed second delete. And whichever you choose, **instrument it**: an endpoint or job that samples cached values against the database and reports disagreement rate turns an invisible race into a number you can act on.
- Does using SET ... NX on the fill path fully close this race?No. NX only prevents a reader from overwriting a value that already exists, so it stops reader-clobbers-writer and slow-reader-clobbers-fresh-reader. In the common delete-only invalidation flow the key is genuinely absent when the slow reader writes, so NX succeeds and the stale value still lands. Closing it properly needs a version comparison or a delayed second delete.
- How does a delayed double delete help, and what does it cost?After committing and deleting the key, a second delete a few hundred milliseconds later removes any stale value written by a reader that was in flight during the first delete. It is simple and needs no coordination, but the delay is a guess, it produces an extra miss, and it depends on a reliable place to run the delayed action — if that process dies, the second delete is silently lost. It reduces probability rather than proving correctness.
- Why can't you just use a plain conditional SET on the value itself?Redis has no compare-the-current-value-and-set primitive for strings, so the comparison has to happen somewhere atomic. Either run a small Lua script that reads the stored version and decides server-side — scripts execute to completion on the single main thread — or use WATCH on the key with MULTI/EXEC, where EXEC aborts if the key was modified after the WATCH. Both give you highest-version-wins instead of last-writer-wins.
- When is it right to accept the race instead of engineering around it?When a stale value for the length of the TTL is cheap. Descriptions, counts, and rankings usually qualify, and a 30-60 second TTL bounds the exposure. Prices, entitlements, balances, and permission decisions do not: for those either use versioned write-back or do not serve the field from cache at all.
A courier dispatched with yesterday's price list arrives after the noticeboard was cleared and pins the old list up — NX is a rule saying 'only pin it if the board is empty', and versioning is stamping each list with a date so nobody pins an older one over a newer one.
saying these in an interview costs you the question
- Claiming write-then-invalidate ordering alone eliminates all staleness races.
- Believing SET NX makes the fill path fully safe, without noticing the key is empty when the race occurs.
- Reaching for a distributed lock around every cache fill as the first answer, ignoring far cheaper mitigations.
- Using wall-clock timestamps from different machines as the monotonic version for compare-and-set.
- Treating the race as impossible because it never reproduced in testing.