When evolving the schema of an event type that's already been written to an append-only event store, which kinds of changes are generally safe to make without breaking existing consumers or replay, and which are dangerous?

## Two audiences read the same log An event-sourced system has two audiences reading the same log over time: the current version of your own code replaying history, and potentially other services or long-running consumers that may run an older or newer schema simultaneously during a rolling deploy. Schema evolution rules exist to let both audiences keep working without a synchronized stop-the-world upgrade. The two properties being protected are: | Property | What it guarantees | |---|---| | **backward compatibility** | new code can still correctly read old events | | **forward compatibility** | old code doesn't blow up when it encounters an event written by newer code, for example during a canary rollout where two consumer versions run side by side | A change is 'safe' precisely when it doesn't put either property at risk. ## The safe category The safe category is **additive-and-optional**. - **Adding a brand-new field to an event**, with a defined default for events written before it existed, is the textbook safe change: old events simply don't have it, and any reader that understands the new schema fills in the default; readers still on the old schema ignore it if the serialization format (JSON, Avro, Protobuf) tolerates unknown fields. - **Adding an entirely new event type** is similarly safe, since it's opt-in. - **Widening a numeric type** (say a 32-bit count to a 64-bit count) is usually safe if the serialization and runtime handle the widened range transparently. The common thread is that nothing that already existed changes meaning; you're only ever adding. ## The dangerous category The dangerous category is anything that changes the meaning, name, type, or cardinality of a field that already has historical data written under the old contract. - **Renaming** `amount` to `total` breaks every reader still looking for `amount`, because the bytes on disk still say `amount` — a classic mistake that looks harmless in code review since it 'obviously' still means the same thing, but breaks any consumer, including your own replay logic, that keys on the field name. - **Changing a field's type** (a string ID becoming a structured object, a float becoming a fixed-point decimal) is dangerous because old serialized values won't parse under the new type without an explicit conversion. - **Removing a field** some reader still depends on, or **making an optional field required**, breaks things for the same reason in reverse. - **Splitting one field into two** (like `total` into `subtotal` + `tax`) is dangerous because a naive deserializer has no way to know how to divide the old value — it needs an explicit rule, i.e. an upcaster. ## The trade-off with additive-only discipline The trade-off with sticking rigidly to 'only additive changes' is that it doesn't cover every real evolution need — sometimes the business genuinely needs to split a field or fix a wrong unit, and there's no additive way to do that. Staying purely additive keeps you safe with zero extra machinery, but you accumulate cruft (deprecated-but-still-present fields, awkward parallel fields) because you can never truly remove or rename anything without a breaking version bump. Doing a breaking change properly (bump version, write an upcaster, deprecate the old field over a compatibility window) costs effort and discipline but keeps the schema clean long-term. Teams that skip that discipline and just rename fields in place accumulate silent data corruption that isn't caught until someone notices a report is wrong months later. ## Failure modes - **In production, the most common failure mode is a 'safe-looking' rename or retype that ships without a version bump;** old events already in the store don't change, so nothing crashes on deploy — instead, every replay of pre-change events either throws a deserialization error (loud, best case) or silently defaults to null/zero for the renamed field (quiet, worst case, compounding through every downstream projection). - **Another common failure is rolling deploys:** if service A starts writing the new event shape before an independent consumer B has deployed code that understands it, B either crashes on the unexpected shape or silently drops data it can't parse — exactly the forward-compatibility risk additive-only changes are designed to avoid. A concrete real-world illustration is Avro's schema evolution rules, used heavily with Kafka: - adding a field is safe **only if it has a default**; - removing a field is safe **only if it had a default in the old schema**; - renaming requires an **explicit alias recorded in the schema itself**, precisely because 'just rename it' silently breaks both old readers and old data with no error at all.