What is the 'copy-and-replace' approach to migrating an event stream, how does it differ from upcasting-on-read, and when would you choose it despite it being more invasive?

## Two ends of one spectrum Upcasting-on-read and copy-and-replace are the two dominant strategies for handling event schema evolution, sitting at opposite ends of a spectrum: | Strategy | The end it sits at | |---|---| | **Upcasting-on-read** | 'do nothing to stored data, translate every time you read it' | | **Copy-and-replace** | 'rewrite the stored data once, then read it plainly forever after.' | **Upcasting-on-read** keeps every historical event byte-for-byte as originally written and inserts a chain of pure transformation functions between storage and the application whenever an old-versioned event is deserialized. **Copy-and-replace** instead treats the migration as a batch ETL job: read the entire existing stream from start to finish, run each event through whatever transformation logic is needed — which, unlike an upcaster, is free to consult external systems or reference data — and append the transformed events into a brand-new stream, typically under a new stream identifier, leaving the original stream intact as an immutable historical artifact. ## What each one can transform The mechanical difference matters because it changes what kinds of transformations are possible. An upcaster is constrained to be a pure function of the event's own payload, because it has to run correctly and identically every single time that event is read, potentially years apart. Copy-and-replace has no such constraint: because it runs once, as a deliberate, supervised operation, it can - join against a currency-rate table as of the event's timestamp, - call a lookup service, - or even have a human review edge cases before the new stream is finalized. This is precisely why copy-and-replace is the right tool exactly where upcasting is not: transformations that need context not recoverable from the event itself. ## Why it exists as a distinct pattern Why this exists as a distinct pattern, rather than always just adding more upcasters: an upcaster chain has a real, compounding cost a one-time rewrite eliminates. Every event ever written keeps needing translation on every future read, so after a system has been through several schema revisions, replaying the oldest events means running them through several chained transformations before domain logic sees them. Copy-and-replace lets a team 'pay down' that accumulated complexity: once the new stream exists and is verified, upcasters for versions with no live stream left can be retired, and future replay reads the current shape directly with zero translation cost. ## The trade-off The trade-off is that copy-and-replace is a bigger operation with real risk on both sides. **On the cost side:** - it requires a maintenance window or a careful dual-write/backfill/cutover dance to avoid losing events written during the migration; - it doubles storage during the transition; - it needs a verification step, often replaying both old-with-upcasters and new-plain and diffing the resulting state; - and unlike upcasting, which is naturally incremental and easy to roll back, a bad copy-and-replace can produce a corrupted new stream that's hard to detect until already in production use. **On the benefit side**, once done, you get a flat, single-version stream going forward with no ongoing translation tax, and the freedom to also restructure the stream's granularity — for example collapsing years of fine-grained events into fewer, richer ones. ## Failure modes at cutover In production, failure modes cluster around the cutover moment and verification gaps. - **If writers aren't paused or dual-written correctly during the batch rewrite,** events written mid-migration can be silently dropped from the new stream, leaving new-stream replay subtly incomplete versus the old stream — the event-sourcing equivalent of a lost-update race condition, easy to miss because both streams still 'work,' just with different content. - **A rewrite that isn't verified against replaying the original stream** can encode the same kind of silent misinterpretation error a bad upcaster would, except now it's baked permanently into the new canonical stream instead of being a fixable transformation function. A well-known real-world pattern for de-risking this, used in systems built around EventStoreDB or Kafka-backed event sourcing, is to write the new stream alongside the old, run consumers against both in parallel for a bake-in period, diff the resulting projections, and only decommission the old stream and its upcasters once confidence is high — essentially a **blue-green deployment applied to the event log itself.**