A distributed key-value store resolves write conflicts with last-write-wins (LWW) based on each replica's wall-clock timestamp. Two data centers each accept a concurrent increment to the same 'view count' key during a brief network partition. When the partition heals, what happens to one of the increments, and how would swapping the counter for a CRDT (conflict-free replicated data type) avoid the problem?
answer
- LWW picks one winner, discards the other
- clock skew can flip causal order
- CRDT merge: commutative+associative+idempotent
- PN-Counter = per-replica inc/dec tallies
- tombstone growth in OR-Sets
basics
~20 sWith LWW, only the "later" timestamped update survives — the other increment is thrown away, silently losing a count. A CRDT counter instead merges both by adding each replica's own tally, so no increment is lost.
solid answer
~30 sLWW picks a single winner per key by comparing timestamps and discards the loser entirely, so if two data centers concurrently increment 5 to 6, LWW keeps only one "6," silently dropping the other increment; clock skew can even let a causally-earlier write "win" over a later one. A CRDT counter (e.g. a PN-Counter) sidesteps this by giving each replica its own increment/decrement tallies and defining merge as combining per-replica counts, so merging is commutative, associative, and idempotent — applying both updates in any order yields 7, with both increments preserved and no timestamp comparison needed.
go deeper
Should grasp that 'last write wins' can throw away a legitimate concurrent update, in plain terms, without needing CRDT vocabulary.
Should explain the LWW mechanism (timestamp comparison, discard loser) and describe at a high level that CRDTs merge rather than overwrite.
Should articulate the CRDT merge properties (commutative/associative/idempotent), name a concrete CRDT type (PN-Counter, OR-Set), and connect clock skew to LWW's silent-loss failure mode.
Should weigh CRDT adoption trade-offs at the system level — metadata growth, type-specific design work, tombstone/GC operational burden — and know real production usage (Riak, Redis CRDB) well enough to justify when LWW is still the pragmatic default.
## What last-write-wins does When two replicas accept writes to the same key while they can't talk to each other — during a network partition, or just because both are serving local writes for low latency — the system ends up with two versions of the truth once they reconnect, and something has to decide what the merged value is. **Last-write-wins (LWW)** resolves this by attaching a timestamp (or a monotonic counter standing in for one) to every write, and when two versions of a key are compared, the one with the larger timestamp is kept and the other is discarded outright. It's attractive because: - it's trivial to implement, - it requires no coordination between replicas, - it always produces a single deterministic value. ## Why "discarded outright" is the problem The problem is exactly that "discarded outright" part: LWW doesn't merge concurrent updates, it picks a winner and throws the loser away, silently. In the scenario given, data center A increments a view count from 5 to 6, and data center B — unaware of A's write because the partition is up — also increments its own local copy from 5 to 6. Neither write is "wrong"; both are legitimate increments that happened concurrently, and the correct merged result is 7. But LWW has no concept of "concurrent" — it only compares timestamps — so when the partition heals, one write's timestamp is picked as the winner, the value stays at 6, and one entire increment vanishes with no error, no conflict flag, and no log entry pointing at what was lost. This gets worse under **clock skew**: if data center B's clock is a few hundred milliseconds fast, a write that happened causally before another write can still carry a later timestamp and incorrectly "win," silently overwriting a more recent, causally-later update. ## How a CRDT fixes it A **CRDT (conflict-free replicated data type)** fixes this at the data-structure level instead of the comparison level: rather than storing a single scalar value and picking a winner, it stores enough structure that merging two concurrent versions is mathematically well-defined and lossless. For a counter specifically, a **PN-Counter** (positive-negative counter) gives every replica its own private increment tally and its own private decrement tally; a replica's visible value is the sum of all replicas' increment tallies minus the sum of all decrement tallies, and merging two replicas' states just means combining each replica's counters (e.g., per-replica max for a simple G-Counter, or summing deltas). Crucially, this merge function is **commutative, associative, and idempotent** — it produces the same result regardless of the order updates arrive in, or how many times a given update is re-applied — so two data centers can each increment independently, exchange their states in any order or even receive stale gossip twice, and always converge to 7, with both increments intact and no coordinator, lock, or timestamp comparison ever required. ## What CRDTs cost The trade-off is that CRDTs aren't free or general-purpose: - **LWW works on any value you can attach a timestamp to** — strings, JSON blobs, arbitrary blobs. - **CRDTs require a purpose-built data type per use case** — counters, sets (G-Set, OR-Set), registers (LWW-Register, MV-Register), maps — and picking the wrong one still loses information (a naive last-write-wins register is itself a CRDT, but it inherits the same lost-update problem as plain LWW for non-mergeable data like "the current shipping address," where there's no sensible way to "merge" two addresses). - **CRDTs also carry more metadata per key** — a PN-Counter needs one integer pair per replica, not one integer — and that metadata grows with the number of replicas or, for sets, can grow with history unless periodically compacted (tombstone growth is a known operational pain point for OR-Sets). ## How it shows up in production In production, teams that adopt plain LWW for something that actually needs merging see it as data that "loses" writes under concurrent load with no visible error — a shopping cart implemented as an LWW value can lose an "add item" that raced with another "add item" from a second tab, and the bug only reproduces under real concurrency, making it notoriously hard to catch in testing. - **Riak** popularized CRDT counters, sets, and maps specifically to fix this class of bug in an AP (available, partition-tolerant) key-value store. - **Redis Enterprise's Active-Active (CRDB)** feature uses CRDTs so multiple regions can accept local writes to the same key and merge without a coordinator. - **Cassandra**, by contrast, defaults to LWW at the cell level and expects application code to model data (e.g., using its native counter type, itself CRDT-like) where plain overwrite semantics aren't safe. The interview-relevant judgment is recognizing that LWW is the right default only when losing a concurrent update is actually acceptable — for anything where every concurrent write carries independent information (counts, set membership, collaborative edits), a CRDT or an explicit merge function is the mechanism that prevents silent data loss.
- Why is a naive LWW-Register itself still considered a CRDT, even though it can lose data?A CRDT only needs its merge function to be commutative, associative, and idempotent so all replicas converge to the same value regardless of message order — LWW-Register's 'keep the higher timestamp' merge satisfies that mathematically. It's a valid CRDT for convergence purposes, but 'converges to a single value' and 'never loses information' are different properties, and LWW-Register only guarantees the former.
- What operational problem does tombstone growth cause in an OR-Set (observed-remove set) CRDT, and how is it typically mitigated?Removing an element from an OR-Set doesn't delete its underlying tags outright — it marks them as removed (a tombstone) so a concurrent add of the same element elsewhere in the cluster isn't accidentally resurrected or lost; over time these tombstones accumulate and bloat storage and merge cost. It's typically mitigated with periodic garbage collection that safely prunes tombstones once all replicas are known to have observed the removal.
- Could vector clocks alone, without switching to a CRDT, fix the lost-increment problem in the view-count scenario?Vector clocks can detect that two writes were concurrent rather than causally ordered, which lets the system flag a conflict instead of silently picking a timestamp winner — an improvement over plain LWW. But detecting the conflict doesn't resolve it: for a counter, the application still needs merge logic that adds both increments together, which is exactly what a CRDT counter provides; vector clocks alone just tell you a conflict exists.
LWW is like two roommates each buying milk while the other's shopping trip is still 'in flight,' and the fridge log only keeps whichever receipt has the later timestamp — so the log shows one carton bought when really two were, and nobody notices until the fridge is unexpectedly empty. A CRDT is like each roommate keeping their own tally of what they bought, with the shared total being the sum of both tallies no matter which order the receipts get filed.
saying these in an interview costs you the question
- Thinks LWW never loses data, just picks 'the correct' newer value
- Doesn't recognize that clock skew can make LWW pick a causally-earlier write as the winner
- Assumes any CRDT can replace any LWW value with no downside
- Can't explain why a CRDT merge must be commutative/associative/idempotent
- Applies LWW-Register semantics to a counter and expects correct sums
- Doesn't mention that CRDTs need type-specific data structures, not a generic wrapper