Explain Avro's writer schema vs reader schema and how schema resolution works during deserialization.

Avro's superpower is that the schema used to **write** data and the schema used to **read** it do not have to be identical — they only have to be *compatible*. This is **schema resolution**. **Definitions:** - **Writer schema**: the schema the producer used to serialize the bytes. It is fixed at write time and travels with the data (on Kafka, via the schema ID embedded in the message). - **Reader schema**: the schema the consuming application expects. It comes from the consumer's own code/version. **Why both are needed:** to decode Avro binary you *must* have the writer schema (the bytes are just values in writer-schema order, with no field names). But the consumer may have evolved to a newer/older schema version. Avro reconciles the two. **Resolution rules (the important ones):** 1. **Records match by name**, recursively, **not by position**. So field order can differ between writer and reader. 2. **Field in reader but not writer** → Avro uses the reader field's `default`. If there is no default, resolution **fails**. 3. **Field in writer but not reader** → Avro reads and **discards** it. 4. **Type promotion** is allowed in a fixed set: int→long→float→double, string↔bytes. 5. **Enums/unions/aliases**: unknown enum symbols can fall back to an enum `default`; `aliases` let a renamed field/record still match. **On Kafka specifically:** the `KafkaAvroDeserializer` extracts the 4-byte **schema ID** from the message, calls the Schema Registry to get the **writer schema**, and caches it. The **reader schema** is determined by config: - With `specific.avro.reader=true`, the reader schema is the schema baked into the generated **SpecificRecord** class on the consumer's classpath. Avro resolves writer→reader, so the consumer can be on a different schema version than the producer. - With GenericRecord (default), there is effectively no separate reader schema — the writer schema is used directly, so you always see exactly what was written. **Edge cases:** - If a producer adds a **required** field (no default) and an old consumer's reader schema lacks it — that's fine (reader ignores it). But if a *consumer* upgrades to a reader schema with a new required field and reads *old* data lacking it, resolution fails unless that field has a default. This is why **adding fields with defaults** is the safe evolution pattern. - Resolution is per-pair; the registry's compatibility checks (BACKWARD etc.) exist precisely to pre-validate that future writer/reader pairs will resolve. **Mental model:** the writer schema says "here is what the bytes mean"; the reader schema says "here is what I want." Avro is the translator that maps one onto the other field-by-field, using defaults to fill gaps.