In a system split across several services, how do you decide which service is the single source of truth for a given piece of data, and what goes wrong when two services both write it?

## Terms - **Owner**: the single service allowed to write a fact and responsible for its invariants. - **Invariant**: a rule that must always hold (an order cannot ship before payment clears; an email must be unique). - **Read model / local cache**: another service's copy of the owner's data, kept fresh from events, used to avoid a synchronous call. - **Dual write**: writing the same logical change to two systems without a shared transaction (DB + message broker, DB + search index). - **Outbox pattern**: within the same database transaction as the state change, insert a row into an `outbox` table; a separate relay reads that table and publishes to the broker. The write and the intent-to-publish commit atomically. - **CDC (change data capture)**: reading the database's own commit log to emit change events, giving the same atomicity property without an outbox table. ## Choosing the owner — practical criteria, in order 1. **Who enforces the invariants?** If a rule must be checked on every change, the checker must own the data; otherwise the rule can be bypassed. This is the strongest signal. 2. **Who is the system of record for the business?** Follow the bounded context that the domain experts treat as authoritative for that concept. 3. **Who writes it most / originates it?** The producer of the data is usually the natural owner; readers are cheap to serve, writers are not. 4. **Lifecycle alignment**: the fact should live with the entity whose lifecycle it follows (created, changed and deleted together). 5. **Blast radius**: prefer the owner whose availability profile matches how critical the write path is. Go **per field** when an entity legitimately spans contexts: 'customer' in CRM, billing and support are usually three different aggregates that share an identifier, not one record with three writers. Model them as separate aggregates linked by ID, each owning its own fields, rather than fighting over one shared row. ## What actually goes wrong with two writers - **Lost update**: A reads value, B reads value, both write; one change vanishes. No error is raised. - **No deterministic winner**: last-write-wins depends on clocks; skew across hosts makes 'last' arbitrary. Logical clocks or version vectors can order writes but still require a merge policy. - **Split invariants**: each writer enforces its own rules, so the combined state can violate rules both thought they were protecting (two services each ensure at most one active subscription; together there are two). - **Unfixable reconciliation**: with one owner, repair means re-derive. With two owners, repair requires a *business decision* per record about which value was intended. - **Diffusion of responsibility**: on-call cannot answer "who is wrong" because both stores are equally authoritative. ## Giving readers what they need without a second writer | Approach | Freshness | Coupling | Failure mode | |---|---|---|---| | Synchronous query to owner | current | runtime dependency on owner's availability and latency | owner down ⇒ caller degraded | | Local read model from events | eventually consistent | build-time contract only | stale reads; needs replay + idempotency | | Cached copy with TTL | bounded staleness | low | stale within TTL window | | Data replication / shared DB | current | very high — schema becomes a shared contract | any schema change breaks consumers | A reader that must *decide* something on stale data needs care: either accept the staleness explicitly (and design compensating actions), or make the owner the decision point via a synchronous call or a request/reply. The common pattern is: read models for display and filtering, owner-side checks for anything that enforces an invariant. ## Dual write vs outbox — why it matters here Even with a single owner, the owner often must update its own DB and tell the world. Naively: ``` db.commit(change) broker.publish(event) // crash here ⇒ event never sent, copies drift forever ``` or the reverse order (event sent, DB rolled back ⇒ copies show a change that never happened). Neither ordering is safe because there is no transaction across the two systems. The fixes: - **Outbox**: `db.transaction { applyChange(); insertOutbox(event) }`, then a relay publishes and marks sent. Delivery is at-least-once, so consumers must be idempotent. - **CDC**: tail the commit log; events are a function of committed state by construction. - **Listen-to-yourself**: publish first to a durable log, then have the owner consume its own event to update state — makes the log the source of truth (event sourcing territory). ## Edge cases and honest exceptions - **Cross-service transactions** are not available; multi-step business operations use **sagas** with compensating actions, keeping each step within one owner. - **Offline/multi-region writers** genuinely cannot have one live owner; then you accept multiple writers and adopt an explicit conflict resolution model — CRDTs, version vectors with a merge function, or human review queues. The point is that the merge rule is *designed*, not accidental. - **Migration windows** temporarily have two writers; make it explicit, time-boxed, one-directional if possible (dual-write with a designated authority and a continuous diff), and delete the old writer on a date. - **Reference data** (country codes, tax tables) is read-mostly and can be replicated everywhere freely — one owner publishes versioned snapshots.