When automatic conflict resolution (like LWW or vector-clock causality checks) cannot safely pick a winner because two writes are truly concurrent and semantically different, systems like Amazon's Dynamo return both versions to the application as 'siblings.' Concretely, how does an application perform a semantic merge on such siblings, using a shopping-cart example, and what does the application have to guarantee for the merge to be safe?
answer
- siblings returned to app
- union merge for cart-adds
- merge must be commutative/idempotent-safe
- defers decision to domain code
- Riak sibling bloat
basics
~20 sWhen a database can't safely decide which of two conflicting writes is 'right,' it hands both versions to the application, and the application's own code — which understands what the data means — merges them into one sensible result, like combining two shopping carts instead of picking one and losing items from the other.
solid answer
~60 sApplication-level (semantic) merge is what happens when a store detects a true conflict — via vector clocks or similar — and, instead of guessing, returns all the concurrent 'sibling' versions to the caller and lets domain-aware code produce the merged result. The classic example is Amazon Dynamo's shopping cart: if two concurrent writes each add a different item to the same cart, the store can't know that 'add' is the intended semantic, so it returns both cart versions as siblings; the application's merge logic recognizes the shape of a cart-add operation and takes the union of items rather than treating either version as authoritative, so no item is lost even though a naive LWW resolution would have dropped one. For this to be safe, the merge function needs to be commutative and idempotent-ish with respect to how the store might re-present siblings (since anti-entropy can re-deliver the same conflict), and the application has to be willing to pay the cost of writing and maintaining a merge function per data type rather than relying on the store to be smart for it.
go deeper
Should understand that sometimes both conflicting writes are kept and a person/application, not the database, decides how to combine them.
Should be able to walk through the cart-union example and explain why picking one write over the other would be wrong there.
Should articulate the commutativity/idempotency requirement on merge functions and the sibling-accumulation failure mode.
Should be able to decide, for a given data model, whether application-level merge, LWW, or a CRDT is the right investment, weighing engineering cost against correctness needs.
## When the store gives up and asks the application Application-level merge is the resolution strategy a system falls back to precisely when automatic detection (vector clocks, version vectors) has correctly identified that two writes are concurrent but has no way to decide which one is 'right,' because rightness depends on what the data means, which only the application knows. Mechanically, the flow looks like this: 1. a client writes value V1 based on some prior version; 2. concurrently, another client writes value V2 based on the same prior version, without seeing V1; 3. when a later read collects versions from replicas, the vector-clock comparison shows neither V1 nor V2 dominates the other, so the store cannot discard either one without risking data loss; 4. rather than picking arbitrarily (which is what LWW would do), the store returns both V1 and V2 to the requesting client, tagged with a merged vector clock that will supersede both once the client writes back a resolved value; 5. the client (or a library embedded in it) runs a merge function specific to that data's shape and writes the merged result, which the causal history then shows as descending from both parents, closing the conflict. ## Why generic resolution cannot decide The reason this exists is that generic conflict-resolution mechanisms — LWW, arbitrary tie-breaking — are correct only when 'newer replaces older' is actually the right semantics for the field in question, and that assumption is false for a large and important class of data: anything representing an accumulation, a set of independent operations, or a structure where two changes can both be legitimate and additive. A shopping cart is the textbook case: 'add item A' and 'add item B' are not two competing readings of the same fact that need a winner, they're two independent operations that should both survive. Generic mechanisms have no way to know this without being told, so the system defers the decision to the one piece of code that does know — the application. ## The trade-off The trade-off is engineering cost and correctness burden shifted onto the application team. Writing a semantic merge function requires understanding the full space of concurrent edits that can occur for that data type and producing a result that is deterministic and ideally commutative: - merging V1-then-V2 must produce the same result as V2-then-V1; - merging an already-merged value with either original parent again shouldn't reintroduce inconsistency, because anti-entropy and read-repair processes can re-present the same conflict to a client more than once. Getting this wrong (a merge function that isn't commutative, or that assumes an ordering) reintroduces the exact class of bugs — silent, order-dependent data loss — that vector clocks were introduced to prevent, just one layer up the stack. In exchange, the payoff is that data that would otherwise be lost under LWW is genuinely preserved, and the resolution logic can encode real business rules ('union of cart items,' 'max of a counter,' 'concatenate a log') rather than an arbitrary timestamp comparison. ## Failure modes 1. **Sibling accumulation and client-side complexity creep** — the characteristic failure mode. If clients don't reliably read-then-write (resolving siblings promptly), the number of concurrent sibling versions for a hot key can keep growing, since every unresolved read-write cycle can spawn another concurrent branch, and every client that reads the key has to be prepared to handle an arbitrary number of siblings, not just two. Riak, which exposed Dynamo-style siblings directly to application developers, became notorious for this — teams that didn't write disciplined merge logic ended up with keys carrying dozens of unresolved sibling versions, degrading read latency and confusing anyone debugging the data. 2. **The escape hatch** — a second failure mode: developers reaching for LWW as an escape hatch once merge logic gets complicated, quietly reintroducing the exact silent-loss problem the sibling mechanism was built to avoid, just for the fields where the merge was 'too hard.' ## The canonical example The canonical real-world example remains Amazon's own description of Dynamo's shopping-cart use case: the paper explicitly justifies exposing siblings to the client, with the cart's 'add item' operations merged by unioning item sets, arguing that in an e-commerce checkout flow, an item erroneously reappearing after being removed (a false 'un-delete' from a stale sibling) is a far cheaper mistake than an item silently disappearing from a customer's cart, which is the failure mode LWW would risk.
- What happens if a client only ever reads the latest single value and never handles siblings — does the merge still happen?No — if the client library or application code doesn't explicitly request and process all sibling versions, it typically only sees one of them (often whichever the driver picks by default or the first returned), and the other concurrent write is effectively lost from that client's perspective even though the store still has it. Some client libraries default to silently picking one sibling unless configured otherwise, which quietly reintroduces LWW-like behavior on top of a system designed to avoid it.
- Why does the merge function need to be commutative rather than just 'correct for the common case'?Because anti-entropy, read-repair, and retried writes mean the same set of concurrent versions can be presented to a merge function more than once and in different orders across different replicas or clients. If merging A-then-B produces a different result than B-then-A, replicas can permanently diverge on what the 'merged' value even is, defeating the entire point of running a merge — the store needs the merge to converge to the same answer regardless of the order operations are observed in.
- Is there a way to get application-level-merge-style correctness without writing a custom merge function for every data type?Yes — this is exactly what CRDTs (conflict-free replicated data types) formalize: they're data structures whose merge operation is provably commutative, associative, and idempotent by construction, so you get correct automatic merging for well-known shapes (sets, counters, maps) without hand-writing merge logic each time. They don't cover arbitrary business semantics, but for the common accumulation/collection patterns they remove most of the custom-merge burden.
It's like two editors independently adding different footnotes to the same paragraph while offline; instead of an editor-in-chief arbitrarily keeping only one footnote, both are handed back to a human who knows the document's intent and combines them into a single paragraph with both footnotes.
saying these in an interview costs you the question
- Thinks the store automatically merges concurrent writes for you
- Proposes picking one sibling by timestamp as 'the merge'
- Doesn't mention merge functions need to be commutative/order-independent
- Unaware that unresolved siblings can accumulate if clients don't merge promptly
- Confuses this with automatic CRDT merging