skip to content

Why must the dedup check-and-record be a single atomic operation, and what bug appears if you use SELECT-then-INSERT or GET-then-SET?

level: middleimportance: should knowfreq 45%

answer

  1. check-then-act => TOCTOU race
  2. two consumers both pass check, both process
  3. fix: UNIQUE INSERT or SET NX (atomic)
  4. catch unique-violation => skip
  5. JVM lock useless across instances

basics

~20 s

If you check 'have I seen this key?' and then separately record it, two concurrent consumers can both pass the check before either records, so both process the message. You need one atomic step: a UNIQUE-constraint INSERT or Redis SET NX that checks and records together.

solid answer

~50 s

Deduplication is a check-then-act: 'is this key present? if not, record it and process.' Split into two operations — SELECT then INSERT, or GET then SET — a **race window** opens between them. Two consumer threads or instances (same partition after a rebalance, or two messages with the same business key on different partitions) can both run the check, both find the key absent, both record, and both process the side effect: a duplicate, exactly what dedup was meant to prevent. The fix is to make check-and-record a **single atomic operation**: a DB `INSERT` against a column with a UNIQUE constraint (the database rejects the second insert with a unique-violation, which you catch as 'duplicate, skip'), or Redis `SET key val NX` (sets only if absent, atomically, returning failure to the loser). Both push the race resolution into the storage engine, which serializes concurrent attempts. Never rely on application-level locks across instances unless they're themselves distributed and atomic.

go deeper

for a junior

Know that checking and recording must be one step, or two consumers can both slip through.

for a middle

Explain the TOCTOU race and fix it with a UNIQUE-constraint insert or SET NX, catching the violation.

for a senior

Identify concrete concurrency sources (rebalance overlap, cross-partition keys) and co-commit the atomic insert with the side effect.

for a principal

Standardize the atomic-dedup primitive and reject application-lock-based 'solutions' in design reviews.

## Dedup is a check-then-act The logical operation is: **if key not seen, then record key and process**. Any check-then-act has a **time-of-check to time-of-use (TOCTOU)** hazard if the check and the act are separate steps that other actors can interleave between. ## The race, concretely Suppose two consumers (or two threads) handle the same dedup key near-simultaneously. With **SELECT-then-INSERT**: ``` T1: SELECT key -> not found T2: SELECT key -> not found (T1 hasn't inserted yet) T1: INSERT key -> ok, process side effect T2: INSERT key -> ok (no constraint), process side effect AGAIN <-- duplicate ``` Both pass the check because neither has recorded the key when the other looks. The result is a **double side effect** — the precise failure dedup exists to stop. The same happens with Redis **GET-then-SET**: ``` T1: GET key -> nil ; T2: GET key -> nil ; both SET ; both process. ``` ## How concurrency arises here - **Rebalance overlap**: during a consumer-group rebalance a partition can briefly be processed by an outgoing and incoming consumer, replaying records. - **Same business key across partitions**: if dedup keys aren't tied to the partition (a business key can appear on multiple partitions or topics), two partition-consumers process them concurrently. - **Multi-threaded listeners**: concurrency within one process. ## The atomic fixes 1. **DB UNIQUE constraint**: declare the dedup key column `UNIQUE` (or PRIMARY KEY). `INSERT` and let the database arbitrate: the first insert wins; the second throws a **unique-violation** (e.g. SQL state 23505 in Postgres, `DuplicateKeyException` in Spring). Catch it and treat as 'already processed → skip the side effect.' The database serializes concurrent inserts on the same key, so exactly one proceeds. 2. **Redis `SET key val NX`**: atomic set-if-absent; the loser gets `nil` and skips. Add `EX` for TTL. Both move the critical section **into the storage engine**, which provides atomicity and isolation that application code can't reliably replicate across instances. ## Anti-patterns - **SELECT-then-INSERT**, **GET-then-SET**, **EXISTS-then-SET** — all non-atomic, all racy. - **Application mutex** (a JVM lock) — only serializes within one process; useless across multiple consumer instances. - **Optimistic 'check, mostly fine'** — under load the race fires often enough to cause real duplicates. ## Tie-in to ordering The atomic insert should be the *same* transaction as the side effect (see the read-process-write ordering question), so 'won the race' and 'did the work' commit together. Atomicity of the check-and-record is necessary but not sufficient; co-committing with the effect completes the safety.

  • How does the DB resolve two concurrent inserts of the same UNIQUE key?
    It serializes them: the first commit (or insert under the index lock) succeeds; the second violates the unique index and raises a unique-violation error, which the app catches and treats as a duplicate to skip.
  • Why isn't a JVM-level synchronized block enough?
    It only serializes threads within a single process. With multiple consumer instances (the normal scaled deployment), each has its own JVM and lock, so two instances can still both process — only a storage-engine-level atomic op or a distributed lock prevents it.

saying these in an interview costs you the question

  • Using SELECT-then-INSERT or GET-then-SET and believing it's safe.
  • Relying on an in-process lock to coordinate multiple consumer instances.
  • Assuming a single partition consumer means no concurrency (rebalances and multi-threading break that).
  • Not catching/handling the unique-violation, letting it crash the listener instead of treating it as a skip.

context

resolving links…