If one canonical store holds the truth, how do you keep derived copies — caches, read models, denormalized tables, generated code — from silently diverging from it?
answer
- one producer per copy
- copies read-only, DO-NOT-EDIT + CI diff
- rebuild must be cheap, deterministic, idempotent
- measure lag; staleness budget as SLO
- reconcile: checksums/counts, then re-derive
basics
~20 sMake every copy generated, never hand-edited; give each one a defined refresh mechanism (regeneration, invalidation, replication, or an event stream); make regeneration cheap and repeatable; and add a check that compares copy against owner and alerts on mismatch.
solid answer
~50 sFour levers, applied together. **(1) Derivation, not duplication** — every copy has an automated producer: code generated from a schema, a materialized view, a cache populated on miss, a projection built from an event stream. **(2) Write protection** — the copy is read-only to everything except its producer; generated files carry a "do not edit" header, denormalized tables are written only by the trigger/consumer, caches are never the write target. **(3) Cheap, deterministic rebuild** — regeneration from scratch must be a routine, idempotent operation, so recovery from any drift is "rebuild the projection" rather than a manual patch; this implies keeping the derivation replayable (event log, versioned schema) and consumers idempotent. **(4) Drift detection** — CI regenerates and fails the build if output differs from what is committed; a reconciliation job periodically compares aggregates or checksums between owner and copy and emits a metric; staleness itself is measured (replication lag, cache age) against an explicit budget. Without (4), the other three degrade quietly.
code
pseudocode · 15 lines// Derived read model: one producer, idempotent, replayable.
onEvent(e): // from the owner's change log
if e.offset <= lastProcessed: return // idempotent on replay
upsertByKey(readModel, e.key, project(e)) // deterministic projection
lastProcessed = e.offset
// Rebuild = routine operation, not an incident response.
rebuild():
newStore = create()
for e in log.from(BEGINNING): apply(newStore, e)
verify(newStore) // counts/checksums vs owner
swapAlias(readModel -> newStore)
// Drift alarm, scheduled.
check(): if checksum(owner) != checksum(readModel): emit("drift", 1)go deeper
Say copies should be generated automatically and never edited by hand, and that caches need an expiry or invalidation rule.
Name the concrete mechanisms — TTL, write-through, explicit invalidation, materialized views, code generation in CI — and the DO-NOT-EDIT plus CI-diff enforcement.
Cover all four levers, especially replayable and idempotent rebuild, and discuss invalidation trade-offs (coupling vs freshness) and reconciliation jobs with mismatch metrics.
Frame staleness as an explicit SLO with a defined degraded behavior, argue for event-log-based derivation so any projection can be rebuilt, and address copy-of-a-copy lag chains and schema-evolution rollout ordering.
## Vocabulary - **Derived copy**: any store of a fact whose value is produced from the canonical owner — cache entry, read replica row, search-index document, denormalized/aggregate table, materialized view, generated SDK, rendered docs, exported report. - **Producer / derivation mechanism**: the automated path from owner to copy — a code generator, a replication stream, a change-data-capture (CDC) pipeline, an event consumer, a scheduled ETL job, a cache-fill on miss. - **Drift**: copy no longer equals what the owner would produce. - **Staleness**: copy is behind the owner by some time window; drift with a bound and an expiry is a *design*, drift without one is a *bug*. - **Idempotent**: applying the same update twice produces the same result as once — required because delivery mechanisms retry. ## The four levers in detail ### 1. Derivation, not duplication Every copy must have exactly one producer and a documented path back to the owner. Ask of any copy: *what regenerates this, and how do I run it?* If the answer is "a person", it will drift. Common producers: | Copy | Producer | Refresh trigger | |---|---|---| | Cache | fill-on-miss / write-through | TTL, explicit invalidation on write | | Read replica | log shipping / streaming replication | continuous, lag-bounded | | Search index, read model | event consumer / CDC | per event, replayable from offset | | Materialized view | database refresh | on commit, on schedule, or incremental | | Client SDK, DTOs, docs | code generator from schema | build step in CI | ### 2. Write protection The copy must be unwritable except by its producer. Mechanisms: `// GENERATED — DO NOT EDIT` headers plus a CI check; database permissions so only the projection's role can write the read model; a code-review rule; physically separate stores so the write path cannot reach the copy. The classic failure is the "just this once" manual patch to a read model to fix a customer complaint. It works, nobody records it, and the next full rebuild silently reverts it — or worse, the rebuild is never run and the copy is now permanently special. ### 3. Cheap, deterministic rebuild If rebuilding a derived store is a scary multi-day operation, drift becomes permanent because nobody dares fix it properly. Design for rebuild from day one: - keep the **input replayable** (retain the event log / source rows; snapshot + tail); - make the projection **deterministic** — same inputs, same output; no `now()`, no random ordering, no reads of other mutable state; - make consumers **idempotent** (upsert by key, or track processed offsets), so replay is safe; - support **rebuild-alongside-then-swap** (build v2 index, verify, flip the alias) so rebuilds are zero-downtime. For generated code, the equivalent is: generation is one command, output is fully overwritten, and hand-written extensions live in separate files (partial classes, subclasses, adapters) so nothing hand-written is ever inside the generated artifact. ### 4. Drift detection Drift you do not measure is drift you find from a customer ticket. - **Build-time**: CI regenerates and `diff`s against the committed artifact; any difference fails the build. This makes "someone hand-edited the generated file" impossible to merge. - **Runtime, cheap**: compare counts/checksums/aggregates between owner and copy on a schedule (e.g. per-tenant row counts and a hash of a sorted key set) and emit a mismatch metric. - **Runtime, exact**: periodic full reconciliation for high-value data (financial balances), producing a diff report and, where safe, auto-repair by re-deriving the affected keys. - **Staleness SLO**: expose replication lag / consumer lag / cache age as a metric with an alert threshold, and decide explicitly what the application does when the budget is exceeded (serve stale with a banner, fail over to the owner, or fail the request). ## Invalidation, the hard part Cache invalidation is where SSoT usually breaks in practice, because the writer must know every copy that must be invalidated — which is coupling in the opposite direction. Options, with trade-offs: - **TTL only**: simplest, no coupling; guarantees bounded staleness, never freshness. Fine for tolerant reads. - **Write-through / write-behind**: writer updates copy; write-behind risks loss on crash. - **Explicit invalidation on write**: fresh, but the writer must enumerate the copies — brittle as copies multiply. - **Event/CDC-driven invalidation**: the owner emits change events, copies subscribe; decouples writer from copy set, adds eventual-consistency lag and requires ordering/idempotency care. - **Versioned keys**: embed a version or content hash in the cache key so a new version simply misses instead of needing eviction. ## Edge cases - **Copy-of-a-copy**: a projection built from another projection multiplies lag and makes rebuild order matter. Prefer deriving everything from the owner or from the same event log. - **Backfill vs live stream**: rebuilding while events continue to flow needs either a replayable log with offsets or a snapshot-plus-tail protocol; otherwise you get holes. - **Deliberate snapshots**: a value copied on purpose to freeze history (the price at order time, the address on a shipped invoice) is *not* a derived copy of a live fact — it is a different fact. Name it that way so nobody 'fixes' it by re-syncing. - **Schema evolution**: when the owner's schema changes, generated copies must be regenerated and deployed in a compatible order; version the contract and support both shapes during rollout.
- A read model has drifted from the owning database. What is your remediation order?Stop the bleeding (find and disable whatever wrote to the copy directly), then re-derive rather than patch: replay from the event log or rebuild the projection into a new store and swap. Patch by hand only as a time-boxed mitigation, and record it so the rebuild is scheduled.
- How do you stop hand-edits to generated code?A DO-NOT-EDIT header plus a CI job that regenerates and fails on any diff. The header alone is advisory; the diff check is the enforcement. Provide a sanctioned extension seam — separate files, subclasses, adapters — so people have a legitimate place to put customisations.
A photocopy is safe as long as everyone knows where the original is kept and copies are reprinted, not annotated. The moment someone writes a correction on a copy, the archive has two originals and no way to tell which is which.