Schema Evolution and Compatibility

The registry compares the submitted schema document against the stored versions of one named lineage, in the direction that lineage's compatibility mode requires. It is a document-to-document check made at registration time, not an inspection of deployed readers or code.

Adding an optional field only asks an old reader to skip bytes it never needed, and its default gives new readers a value when old writers omit it. Making a field required invalidates every message already written without it.

Identity decides. Where a reader matches fields by number, a rename changes only the label. Where it matches by name, a rename is a delete plus an add, so readers on the old schema quietly stop finding the value.

Three placements are common: a version field inside the payload, a negotiated media type, and a version segment in the address. Each moves the cost elsewhere - routing and caching, client effort, or resource identity - and none removes it.

The direction that holds decides the order. A backward-compatible change lets the readers go first, since new readers cope with data still written under the old schema; a forward-compatible change lets the writers go first. Full compatibility means any order works.

A registry gates a publish only when the decision is server-side, the name being published to has a mode configured, no write path reaches the bytes without registering, and the mode itself is not freely editable by the team being gated. Miss any one and it is a library.

Expand-and-contract ships one incompatible change as a sequence of compatible ones: add the new form beside the old, write both, move readers to the new with a fallback, backfill what already exists, stop writing the old, then remove it.

Each side writes on one hop and reads on the other, so a single deployment imposes opposite requirements at once: an old callee must read new requests while a new caller must read old responses. Only both directions cover both hops.

Whether clients still need a resolution they have not already cached. Warm clients publishing and reading known schema versions keep running; cold starts, a first sighting of a new version, and register-on-publish producers all need the registry and stall or fail without it.

Only a widening whose on-wire byte form is unchanged survives, and only while writers stay inside the old range. Narrowing, signed-to-unsigned reinterpretation and any edit that changes the byte form corrupt values silently rather than failing.

A retired number and name still live in deployed readers and in bytes already written. Reserving them stops a later edit from handing an old reader a different field under an identifier it already believes it understands.

What doubles is the surface: contracts, tests, docs, client support and every later change, which must land in both. What does not fork is the data and the behaviour behind them, so a change of meaning reaches old callers however many shapes you serve.

Match the guarantee to how long data and stragglers outlive a release. A long-retention stream needs a transitive guarantee so current readers can replay all of history; a short-lived channel drained between releases can run honestly on the weaker per-step form.

The guarantee stops being a property of the platform and becomes a per-name, per-day unknown. Consumers can no longer reason about what is enforced anywhere, the cost of an override lands on teams who never made it, and the evidence usually disappears with the setting.

Price a one-time risk against a permanent cost. An in-place edit is available only when the layout is unchanged, the direction is safe and the readers are enumerable; a replacement field burns identifiers and adds precedence logic forever, but makes the ambiguity visible per message.

Set the window from the caller population's slowest realistic cycle, not from habit; signal it both machine-readably in responses and through a human channel; and decide the shut-off from per-caller usage measured over a full cycle, escalating through warnings and brownouts rather than a single date.

Transitive compatibility quantifies over every earlier version, not just the immediately preceding one. Plain per-step checks do not compose: version 3 can pass against version 2, and version 2 against version 1, while a version 3 reader still fails on version 1 data.

A new version of the shared type, then a re-registration of every referencing schema that is to adopt it — each re-checked against its own history. A reference pins a version, so nothing moves until its owner moves it, and the shared type's own verdict says nothing about the dependants.

The value usually arrives intact, so the decode is not where it breaks. It breaks in application logic that branches over the members it knows and has no arm for this one — an additive schema edit that is not additive for consumers.

Only a self-carried marker works: the version must sit in the bytes, because negotiation needs a counterpart to ask and an address to ask it at, and a consumer reading a queued message or an archived file has neither.