An optimistic check-and-set loop against Redis works fine in staging but in production one popular key sees so much concurrent traffic that most attempts are cancelled and the service latency rises. How do you diagnose and fix this?

## Why the loop looks fine until it does not Optimistic concurrency has a non-linear failure mode. With conflict probability *p*, expected attempts are about 1/(1−p). At p=0.05 you retry once in twenty operations and nobody notices. At p=0.6 you average 2.5 attempts, each costing a read and a transaction setup, and — crucially — the extra load you generate *raises* p further, because more clients are now sitting in the window between WATCH and EXEC. That positive feedback is why the graph looks flat and then goes vertical. Staging, with one tenth the traffic, sits on the flat part. ## Step 1 — measure the right things Three numbers make the diagnosis unambiguous: - **Attempts per successful operation**, as a distribution, not a mean. A p99 of 8 with a mean of 1.3 means a small set of keys is starving while everything else is fine. - **Write rate on the watched key**, and its sources. Watch invalidation counts *any* write, so a job rewriting an unchanged value at 500/s is indistinguishable from 500 real updates. Audit who writes the key before you redesign anything. - **Key skew.** `redis-cli --hotkeys` (with an LFU policy) or sampled key metrics tell you whether this is one hot key or a broad load problem. The fixes are completely different. Also check granularity: if you are watching a hash and the invalidating writes touch a field your logic never reads, the whole problem is an artefact of key layout. ## Step 2 — prefer removing the conflict over managing it **Atomic command.** The cheapest fix is discovering the operation already exists: counters (`INCRBY`), claim-once (`SET k v NX EX`), monotonic maxima (`ZADD key GT`), conditional set (`SET k v XX`), take-and-replace (`SET k v GET`, `GETDEL`). Zero conflicts, one round trip, no code. **Server-side script.** When the decision is real logic, one Lua script or Redis Function performs read, decision and write with nothing interleaved. Conflicts stop existing; clients simply serialise on the execution thread, which is bounded and fair, instead of racing and discarding work. The constraint is duration: while the script runs nothing else does, so it must be short and loop-free over unbounded collections. This converts wasted work into orderly queuing — usually the single biggest win available. **Shard the key.** For aggregate structures, split `counter` into `counter:{shard}:0..N-1`, have each writer pick a shard (by client id or at random) and have readers sum them. Contention per key falls by roughly N; the cost is a fan-out on read and a slightly more complex model. The same trick applies to sets and to rate limiters. In a cluster, decide deliberately whether the shards share a hash tag (co-located, aggregatable in one script) or spread across nodes (more throughput, aggregation done client-side). **Cut the write rate.** Coalesce: buffer updates in the client for 50 ms and write once. Move volatile fields out of the watched key so refreshers no longer invalidate readers. Question whether the write needs to happen synchronously at all. ## Step 3 — make the loop survivable regardless Even with the right design, keep the guardrails: - **Bounded attempts.** After N tries, fail fast and surface a specific error rather than looping. An unbounded loop under a traffic spike becomes a self-inflicted denial of service, consuming connections and CPU on both ends. - **Jittered backoff.** Without randomisation, all conflicting clients retry in lockstep and re-collide; jitter spreads them out and is often worth several points of success rate on its own. - **Load shedding.** When retries exceed a threshold, shed or queue the work; it is better to reject 2% of requests quickly than to melt the instance for everyone. - **Release watches on every path.** In pooled clients, an early return that skips EXEC/DISCARD/UNWATCH hands the next borrower stale watches and mysterious nil results. ## Step 4 — reconsider the model Sometimes the honest answer is that the hot key is a design smell: a global counter that every request touches, a single sorted set acting as a work queue, a shared config blob rewritten on every deploy. Ask whether the value must be exact and immediate. Approximate counters, per-partition aggregates, or a stream that a single consumer folds into the aggregate remove the contention entirely rather than distributing it. ## What a strong answer sounds like Lead with measurement, name the feedback loop that makes optimistic CAS fail suddenly, then propose fixes in order of how much contention they *remove* rather than manage — atomic command, server-side script, sharding, write-rate reduction — and finish with the guardrails and the question of whether the hot key should exist at all.