skip to content

For an Avro schema, which specific field changes are allowed under BACKWARD, FORWARD, and FULL, and what role do default values play?

level: seniorimportance: must knowfreq 60%

answer

  1. BACKWARD: delete free, add needs default
  2. FORWARD: add free, delete needs default
  3. FULL: only add/remove fields WITH default
  4. Default = value reader uses when field is absent
  5. Rename needs an Avro alias; type change usually breaks

basics

~20 s

BACKWARD allows deleting fields and adding fields that have a default. FORWARD allows adding fields and deleting fields that have a default. FULL allows only adding or removing fields that have a default. Defaults let the new schema fill in missing data.

solid answer

~50 s

Defaults are the hinge. Under BACKWARD, the new (reader) schema must cope with old data: you may delete a field (the reader just ignores the extra bytes' absence) and you may add a field only if it has a default, so the reader can supply a value for old records that lack it. Under FORWARD, the old (reader) schema must cope with new data: you may add a field (old readers ignore the unknown field) and you may delete a field only if it had a default, so old readers can fill in the now-missing field. FULL is the intersection: you may only add or remove fields that have defaults, because that single change must read cleanly in both directions. Changing a field's type, or renaming it without an Avro alias, is incompatible in all modes. Aliases let a rename be treated as the same field for resolution.

code

json · 9 lines
json
{
  "type": "record",
  "name": "User",
  "fields": [
    { "name": "id", "type": "long" },
    { "name": "email", "type": ["null", "string"], "default": null },
    { "name": "fullName", "type": "string", "aliases": ["name"] }
  ]
}

go deeper

for a junior

Know that adding optional (defaulted) fields is the safe baseline change.

for a middle

Map add/remove × default to each of BACKWARD/FORWARD/FULL correctly.

for a senior

Explain WHY via reader/writer resolution and handle renames via aliases and enum defaults.

for a principal

Generalize the reader/writer logic across Avro/Protobuf/JSON Schema and codify evolution conventions for the org.

## Setup Avro resolves data using a **writer schema** (used to encode) and a **reader schema** (used to decode). When they differ, Avro applies **schema resolution rules**. Compatibility modes are just a guarantee that these resolution rules will succeed for the reader/writer pairing the mode protects. The pivotal rule: if the reader expects a field the writer's data does not contain, Avro uses the field's **default**; if the writer's data contains a field the reader does not expect, Avro **ignores** it. ## BACKWARD (new reader reads old writer) - **Delete a field**: SAFE. Old data has the field's bytes; the new reader simply doesn't have that field, and Avro ignores the extra. No default needed. - **Add a field**: SAFE ONLY IF it has a `default`. Old data lacks the field; the new reader needs the default to fill it in. Without a default, decoding old records fails. ## FORWARD (old reader reads new writer) - **Add a field**: SAFE. New data has the field; the old reader doesn't know it and ignores it. No default needed on the old side (the new schema may or may not have one). - **Delete a field**: SAFE ONLY IF the deleted field had a `default` in the old (reader) schema. New data lacks it; the old reader fills from its default. Without a default, the old reader can't decode new records. ## FULL (both directions) - The only changes that are simultaneously BACKWARD- and FORWARD-safe are **adding a field with a default** and **removing a field that has a default**. Anything else breaks one of the two directions. ## Defaults — the unifying idea A **default** is the value the reader substitutes when the writer's data omits a field. Adding-with-default protects readers of old data (BACKWARD); the deleted field's prior default protects readers that still expect it (FORWARD). This is why "always give fields a default" is the standard advice for safe evolution. ## Other change types - **Type change**: generally incompatible. Some numeric promotions are allowed by Avro resolution (e.g. int→long, float→double; the reverse is not), but type changes are risky and usually rejected. - **Rename without alias**: incompatible — Avro treats it as delete-old + add-new, which usually violates the add-needs-default / delete-needs-default rules. - **Aliases**: Avro's `aliases` attribute lets a reader match a renamed field (or record/enum) to the old name during resolution, making an otherwise-breaking rename compatible. - **Enums**: adding a symbol can break FORWARD unless the enum has a default symbol (Avro 1.9+); removing a symbol can break BACKWARD. ## Cross-format note Protobuf and JSON Schema have their own rules (e.g. Protobuf's field-number reservation, JSON Schema's `additionalProperties`/`required`), so "defaults make it safe" is an Avro-centric phrasing. The reader/writer-direction logic, however, is the same across formats.

  • You need to rename an Avro field without breaking BACKWARD compatibility. How?
    Add the new field name as the canonical name and list the old name in the field's `aliases`, so Avro resolution maps old data's field to the renamed reader field.
  • Why can't you add a field WITHOUT a default under BACKWARD?
    Old records don't contain that field's bytes; the new reader has no value to assign and no default to fall back on, so decoding the old record fails.

saying these in an interview costs you the question

  • Saying adding a field is always safe regardless of mode — it needs a default under BACKWARD.
  • Claiming defaults are irrelevant to compatibility.
  • Asserting a rename is fine without an alias.
  • Stating FULL allows arbitrary add/remove — only with defaults.
  • Confusing the directions: under FORWARD it's DELETE that needs the default, not add.

context

resolving links…