skip to content

What is Apache Avro and why is it commonly used as the serialization format for Kafka records?

level: juniorimportance: must knowfreq 70%

answer

  1. schema + compact binary
  2. byte[] contract for producer/consumer
  3. schema ID, not full schema, in message
  4. evolution via registry compatibility
  5. smaller than JSON

basics

~20 s

Avro is a compact binary serialization format that stores data with a separate schema. With Kafka it gives small messages, a typed contract for producers and consumers, and safe schema evolution via a Schema Registry.

solid answer

~40 s

Apache Avro is a binary serialization framework where every record is described by a schema (an Avro `.avsc` JSON document or Avro IDL). The schema is not stored in each message; instead the Confluent `KafkaAvroSerializer` registers the schema in a Schema Registry and writes only a small schema ID plus the compact binary payload. This makes messages much smaller than JSON and gives producers and consumers a shared, typed contract. Because the schema is explicit and the registry enforces compatibility rules (BACKWARD, FORWARD, FULL), Avro supports safe schema evolution: producers can add fields without breaking older consumers. Avro also supports rich types, defaults, and logical types (decimal, timestamp). This is why Avro (alongside Protobuf and JSON Schema) is one of the three formats Confluent Schema Registry natively supports.

go deeper

for a junior

Know that Avro = compact binary + a schema, and that it gives a typed contract and small messages on Kafka.

for a middle

Explain the schema-ID wire format and that the registry stores the schema, not the message.

for a senior

Tie Avro choice to evolution/compatibility guarantees and operational cost of the registry dependency.

for a principal

Reason about Avro vs Protobuf vs JSON Schema trade-offs org-wide and the governance model around schemas.

**Serialization** means turning an in-memory object into bytes to send over the wire; **deserialization** is the reverse. Kafka itself only moves opaque `byte[]` keys and values, so producers and consumers must agree on how those bytes are encoded. **Apache Avro** is a serialization framework from the Hadoop ecosystem. Its defining trait is that data is always paired with a **schema**. A schema is a JSON document (file extension `.avsc`) or an Avro IDL file that declares the fields, their types, defaults, and documentation. Example `.avsc`: ```json { "type": "record", "name": "User", "namespace": "com.acme", "fields": [ {"name": "id", "type": "long"}, {"name": "email", "type": ["null", "string"], "default": null} ] } ``` **Why Avro fits Kafka:** 1. **Compactness** — Avro binary encoding writes field *values* in schema order with no field names or tags, so messages are far smaller than JSON. This matters at Kafka's throughput. 2. **A typed contract** — both sides share the schema, so a consumer knows exactly what fields and types to expect, instead of parsing free-form JSON. 3. **Schema evolution** — Avro was designed so a reader can read data written with a *different but compatible* schema. Combined with the **Confluent Schema Registry**, you get enforced compatibility rules so a producer change can't silently break consumers. **How it works on Kafka with Schema Registry:** instead of embedding the full schema in every message (wasteful), the `KafkaAvroSerializer` registers the schema once with the registry, gets back an integer **schema ID**, and writes a 5-byte header (a magic byte `0x0` + 4-byte schema ID) followed by the Avro binary payload. The `KafkaAvroDeserializer` reads the ID, fetches the schema from the registry (cached), and decodes. **Edge cases / trade-offs:** Avro requires the schema to decode — you cannot read the bytes without it (unlike self-describing JSON). The registry becomes an operational dependency. Avro's binary form is not human-readable, which complicates ad-hoc debugging; tools like `kafka-avro-console-consumer` exist for this.

  • Why is the full schema not embedded in every Kafka message?
    Embedding it per-message would bloat every record. Instead the serializer registers the schema once and writes a 4-byte schema ID; consumers resolve the ID against the Schema Registry and cache it.
  • What are the alternatives to Avro that Confluent Schema Registry supports?
    Protobuf and JSON Schema. All three share the same registry, schema-ID wire format, and compatibility checking; they differ in encoding and tooling.

saying these in an interview costs you the question

  • Saying the full Avro schema is shipped inside every Kafka message
  • Claiming Avro bytes are self-describing and readable without the schema
  • Confusing Avro (the format) with the Schema Registry (the service that stores schemas)

context

resolving links…