Describe how to guard an expensive cache recompute with a Redis mutex based on `SET lock:<key> <token> NX PX <ttl>`. What should a request that fails to acquire the lock do, and how is the lock released safely?
answer
- SET NX PX in one command — never SETNX then EXPIRE
- Unique token + Lua compare-and-delete on release
- PX > p99 recompute; watchdog extends if unbounded
- Losers: stale > bounded poll > degrade; never unbounded wait
- Best-effort lock: failure costs duplicate work, not correctness
basics
~20 sOn a miss, try SET lock:key <random token> NX PX <ttl>. The winner recomputes, writes the cache, then releases with a Lua script that deletes only if the stored value still equals its token. Losers serve stale, poll briefly, or degrade — they must not queue unbounded.
solid answer
~1 minOn a cache miss: 1. `SET lock:<key> <uuid> NX PX <ttl>`. `NX` makes it set-if-absent and `PX` gives it an expiry, atomically — the old `SETNX` then `EXPIRE` pair is not atomic and leaves an immortal lock if the client dies in between. 2. The winner recomputes, writes the value with `SET key value EX ...`, then **releases with a compare-and-delete Lua script**: delete only if the stored value equals its own token. A plain `DEL` can erase a lock that already expired and was re-acquired by someone else. 3. Losers pick a strategy: return a stale copy if you keep one (best), poll `GET key` every 20–50 ms up to a bounded deadline, or return a degraded response. Never block indefinitely — that converts a database stampede into a thread-pool stampede. Set `PX` above the p99 recompute time; too short and a second worker starts while the first is still running, too long and a crashed holder blocks refreshes for that whole period. A watchdog can extend it while work continues. Crucially this is a *best-effort* lock guarding a performance optimisation: worst case on failure is duplicate work, not corruption — so the correctness debates around distributed locking do not apply here.
code
text · 7 lines# acquire (atomic set-if-absent with expiry)
SET lock:product:123 "9f2c...uuid" NX PX 5000
# -> OK (winner)
# -> (nil) (loser)
# winner writes the value, then releases with the script below
SET product:123 "{...}" EX 300go deeper
Know the command SET lock:key value NX PX ttl and that exactly one caller gets OK and does the work.
Explain why NX+PX must be one command, why the value should be a unique token, and what the losing callers do.
Cover the compare-and-delete release, TTL sizing against p99 recompute, watchdog extension, bounded waiting versus stale serving, and releasing on failure.
Position it as a best-effort optimisation with an explicit failure budget: define acceptable duplicate work, decide the degradation behaviour under contention, and rule out lock-protected side effects that would demand real distributed-locking guarantees.
## The shape of the solution The stampede exists because nothing coordinates the concurrent misses. A mutex adds exactly that coordination, using Redis itself as the rendezvous point. ``` value = GET key if value: return value token = random() if SET lock:key token NX PX 5000: # I won try: v = recompute() SET key v EX 300 return v finally: release(lock:key, token) # compare-and-delete else: # I lost return stale_or_wait_or_degrade() ``` ## Why `SET … NX PX` and not `SETNX` + `EXPIRE` `SETNX` sets a key only if it does not exist; `EXPIRE` attaches a TTL. Issued as two commands they are not atomic from the client's point of view: if the process crashes, the connection drops, or the node fails over between them, the lock key exists **with no expiry**. Nothing will ever delete it, and that cache entry can never be refreshed again — a permanently stuck value that usually surfaces days later as "why is this page frozen". `SET key value NX PX ttl` performs both in one command (available since Redis 2.6.12) and is the only form you should write. ## The token, and why release must be compare-and-delete Give every acquisition a unique random value (a UUID, or a counter plus process id). Release must be: ```lua if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) else return 0 end ``` The failure this prevents: worker A takes the lock with `PX 5000`, but its recompute takes 7 seconds. At t=5 s the lock expires; worker B acquires it and starts its own recompute. At t=7 s worker A finishes and issues a bare `DEL lock:key` — deleting **B's** lock, so worker C immediately acquires it, and now three workers are recomputing. The bare `DEL` turns a bounded overlap into an unbounded one. A Lua script runs atomically on the server, so the read and the delete cannot interleave with another client. (`GETDEL` does not help — it is unconditional. Redis 7's `SET … IFEQ`-style conditional forms and Functions are alternatives, but the Lua compare-and-delete is the portable idiom.) ## Choosing the lock TTL The TTL is the answer to "how long may a crashed holder block refreshes?" and simultaneously "how long before a slow holder gets overlapped?" - Too short: overlapping recomputes, the very thing you built this to prevent. - Too long: if the holder dies, the value stays stale (or absent) for the entire TTL, and every request in that period must fall back. Start at roughly p99 recompute latency plus a margin. If that latency is unbounded or highly variable, add a **watchdog**: a timer in the holder that periodically extends the lock (`PEXPIRE` guarded by the same compare-token check, or a Lua extend script) while the work is genuinely in progress, and stops on completion or failure. Never extend blindly — a hung worker would then hold the lock forever. ## What the losers do — the decision that matters most This is where designs succeed or fail. - **Serve stale (best).** If you kept the previous value alive — physical TTL longer than logical freshness — losers return it instantly. No waiting, no thread occupancy, no user-visible latency. This is why stale-serving and the mutex are usually built together. - **Poll with a deadline.** Sleep 20–50 ms, `GET key` again, repeat up to a hard cap (e.g. 500 ms or ~10 attempts), then degrade. Bounded, simple, and it keeps the caller's latency predictable. Jitter the sleeps so waiters do not wake in lockstep. - **Degrade.** Return a placeholder, an empty list, HTTP 503 with `Retry-After`, or a cached-at-a-coarser-granularity answer. Explicitly the right choice under extreme load. - **Recompute anyway (wrong).** Some implementations let losers proceed to the source "just this once". That reintroduces the stampede. - **Block indefinitely (wrong).** Unbounded waiting moves the pile-up from the database into your request threads and connection pools, and takes down endpoints that have nothing to do with this key. ## Why the Redlock debate does not apply A cache mutex is an **optimisation, not a correctness mechanism**. If two workers briefly hold it, the outcome is duplicate work and one redundant `SET` of the same value — wasteful, not wrong. That is why a single-instance `SET NX PX` with a token is entirely adequate here, and why the arguments about clock drift, fencing tokens and multi-node lock algorithms — which matter when a lock protects a non-idempotent side effect — are out of scope. Do check that your recompute *is* idempotent and side-effect-free; if acquiring the lock also triggers, say, a payment or an email, you no longer have a cache-stampede problem, you have a distributed-locking problem with different rules. ## Operational details - **Namespace** the lock (`lock:<key>`) so it never collides with the value, and give it a much shorter TTL than the value. - **Failure path:** if the recompute throws, release the lock in a `finally` so the next request can retry immediately instead of waiting out the TTL; consider a short negative-cache entry so a persistently failing source does not get hammered by successive winners. - **In cluster mode**, the lock key and the value key may live in different slots; that is fine for separate commands, but if you ever want a single Lua script over both, tag them (`{product:123}`) so they share a slot. - **Metrics:** lock acquisitions, contention (losses), waiter timeouts, and recompute duration. Contention approaching request rate means your TTL or refresh strategy needs work, not your lock.
- Why release the lock with a Lua script instead of a plain `DEL`?Because the lock you are deleting may no longer be yours. If your recompute outran the lock's `PX`, the key expired and another worker acquired it; a bare `DEL` erases that worker's lock and lets a third worker in, so overlap compounds instead of being bounded. The Lua script compares the stored token to yours and deletes only on a match, and it runs atomically on the server so nothing can interleave between the read and the delete.
- What happens if you set the lock's `PX` shorter than the recompute takes?The lock expires mid-flight, a second worker acquires it and starts a duplicate recompute, and the first worker may then delete the second's lock unless you use compare-and-delete. You get exactly the duplicate work the lock was meant to prevent, with extra confusion. Size `PX` above p99 recompute latency, or add a watchdog that extends the lock while the work is verifiably still running.
- Under heavy contention, is it better for losers to wait for the lock or to return stale data?Return stale data whenever you have it: the request completes immediately, no thread is held, and the staleness is bounded by the refresh interval. Waiting simply relocates the pile-up from the database into your application's thread pools and connection pools, where it can take down unrelated endpoints. If no stale copy exists, poll with a hard deadline and jittered backoff, then degrade.
saying these in an interview costs you the question
- Using `SETNX` followed by `EXPIRE`, which can leave a lock with no expiry if the client dies in between
- Releasing with a bare `DEL`, which can delete a lock another worker now holds
- Letting lock losers fall through to the database anyway
- Making waiters block without a deadline, moving the stampede into the application's thread pool
- Treating the cache mutex as a correctness-grade distributed lock and adding Redlock complexity it does not need
- Setting the lock TTL by habit rather than from measured recompute latency