How would you implement a Redis-based dedup store with SETNX, and what TTL/cleanup and durability concerns must you handle?

## The Redis primitive The modern, correct command is a single atomic call: ``` SET <dedupKey> 1 NX EX 86400 ``` - **NX** — *Not eXists*: perform the set only if the key is absent. Returns the value/OK on success, `nil` if the key already existed. - **EX 86400** — set a **TTL** of 86400 seconds (1 day); Redis auto-deletes the key when it expires. (The legacy `SETNX` command does the NX part but can't set a TTL atomically — combining `SETNX` then `EXPIRE` has a race if you crash between them. Always use `SET ... NX EX`.) Decision logic: - **OK returned** → first time seeing this key → process the message. - **nil returned** → duplicate → skip. The atomic NX is what makes this safe under **concurrency**: if two consumer threads/instances race on the same key, exactly one gets OK. ## TTL and cleanup The TTL is your cleanup mechanism — no separate sweeper needed (unlike a SQL table, which needs a scheduled DELETE). Sizing: - **Too short**: an old duplicate arriving after expiry is re-admitted and re-processed. The TTL must exceed the **maximum realistic duplicate window**: consumer lag spikes, retry/backoff durations, manual replays, and dead-letter-queue reprocessing horizons. - **Too long**: more keys held in memory → higher RAM cost. Redis keys are small but billions of them aren't free; estimate `peak_msg_rate * TTL` for worst-case key count. ## Durability concerns (the Redis-specific risk) Redis is **in-memory** with optional persistence: - **RDB snapshots** — periodic; a crash loses everything since the last snapshot. - **AOF** — append-only log; with the common `appendfsync everysec` setting, up to ~1 second of writes can be lost on crash. Either way, a crash can **lose recently written dedup keys**, re-admitting their messages as 'new' → duplicate side effects. This is acceptable only if your side effect tolerates rare duplicates. For stricter needs use AOF `appendfsync always` (slower) or, better, don't rely on Redis alone. ## The cross-system atomicity gap `SET NX` and the side effect are in **different systems** (Redis + your effect). Crash windows: - Crash after `SET NX` OK but before the side effect → key marked done, effect never happened → **lost work** on redelivery (the dedup skips it). - To mitigate: write an 'in-progress' marker first and only finalize after the effect, with a reconciliation/timeout to recover orphaned markers — more complex and still imperfect. This is why, when the side effect is a DB write, a **DB UNIQUE constraint co-committed with the effect** is strictly safer than Redis dedup. Choose Redis when throughput/latency dominate and rare gaps are acceptable, or when the side effect itself lives in Redis. ## Operational notes - **Key naming**: namespace it, e.g. `dedup:orders:{orderId}`, to avoid collisions across consumers. - **Clustering**: in Redis Cluster, a single key maps to one slot — fine; no cross-slot transaction needed for one key. - **Memory eviction**: ensure the dedup keyspace isn't subject to an LRU/LFU `maxmemory-policy` that could evict non-expired keys, silently re-admitting duplicates. Use a policy like `noeviction` or isolate dedup data.