Schemas and Serialization

Schema Registry has BACKWARD (default), FORWARD, FULL, NONE, and a _TRANSITIVE variant of each. They control what schema changes are allowed: BACKWARD lets new consumers read old data, FORWARD lets old consumers read new data, FULL means both, NONE disables checks.

Deserialization turns raw bytes back into objects. A Kafka topic is just bytes from whoever produced them, so a malicious producer can send crafted bytes that exploit the consumer's deserializer — for example triggering code execution or crashing it.

A poison pill is a record the consumer cannot deserialize (corrupt or wrong-format bytes). Deserialization happens inside poll(), so it throws every time, the offset never advances, and the consumer is stuck retrying the same record forever.

Use the format-specific serializer: KafkaProtobufSerializer or KafkaJsonSchemaSerializer instead of KafkaAvroSerializer. Each subject in the registry carries a schemaType (AVRO, PROTOBUF, or JSON) so the registry knows which format the stored schema is.

It is a separate service that stores message schemas (e.g. Avro) and gives each one a unique ID. Producers register a schema and write its ID into the message; consumers fetch the schema by ID to deserialize, so both sides agree on the data shape.

A Serializer turns a Java/Kotlin object into a byte[] so Kafka can store and send it. Producers pick one via the key.serializer and value.serializer config properties — one for the key, one for the value.

A subject is the named scope under which schema versions are registered and evolution is checked. The default TopicNameStrategy names it after the topic plus a suffix: <topic>-key for keys and <topic>-value for values.

GenericRecord is a schema-driven, map-like object you read by field name with no codegen. SpecificRecord is a generated Java class with typed getters. Setting specific.avro.reader=true tells KafkaAvroDeserializer to return the generated class.

BACKWARD: upgrade consumers first, then producers — new consumers can read both old and new data. FORWARD: upgrade producers first, then consumers — old consumers can still read the new data the upgraded producers write.

Schema formats parse bytes into a fixed, known data shape and never let the payload decide which classes to create. Native Java serialization reconstructs arbitrary object graphs and runs class logic, which attackers exploit for code execution.

ErrorHandlingDeserializer wraps your real deserializer. If the delegate throws, it catches the error, returns null, and stashes the exception and raw bytes in record headers. poll() then succeeds, so a DefaultErrorHandler/DeadLetterPublishingRecoverer can route the bad record to a DLQ and advance the offset.

A subject is a named scope (usually one per topic, like orders-value) under which schemas are registered and versioned (1, 2, 3...). A global schema ID is a single number that uniquely identifies one physical schema across the whole registry, independent of subjects and versions.

Kafka bundles StringSerializer, ByteArraySerializer, IntegerSerializer, LongSerializer, DoubleSerializer, ShortSerializer, FloatSerializer, and UUIDSerializer (plus ByteBuffer/Bytes/Void). Numbers use fixed-width big-endian bytes; String/UUID use a configurable charset (UTF-8 default).

Built-in serializers return null bytes for null input — they don't crash. A record with a non-null key but a null value is a tombstone: on a log-compacted topic it signals 'delete this key,' and compaction eventually removes it.

The writer schema is the one used to encode the bytes; the reader schema is the one the consumer wants to decode into. Avro resolves the two by matching fields by name, applying defaults for missing fields and dropping unknown ones.

BACKWARD allows deleting fields and adding fields that have a default. FORWARD allows adding fields and deleting fields that have a default. FULL allows only adding or removing fields that have a default. Defaults let the new schema fill in missing data.

Default typing lets the JSON itself name a Java class to instantiate, so an attacker can name a 'gadget' class that does something harmful when built — potentially code execution. Remediate by disabling default typing and deserializing into fixed, known types.

After the 5-byte magic+schema-ID header, KafkaProtobufSerializer writes a message-index: a varint-encoded array that points to which message type inside the .proto file the payload uses. Then the Protobuf binary body follows. It is needed because one .proto can declare many messages.

schema.registry.url tells the serializer/deserializer where the registry lives. auto.register.schemas, when true (default), makes the producer automatically register any new schema it sees. In production it's often set to false so schemas are registered deliberately (e.g. in CI), preventing accidental or incompatible schemas.

The serializer writes 1 magic byte (0x0), then a 4-byte big-endian schema ID, then the serialized payload. The consumer reads the magic byte, reads the ID, fetches that schema from the registry (cached), and decodes the rest of the bytes.

RecordNameStrategy names the subject after the schema's fully-qualified record name, so it's topic-independent and shared across topics. TopicRecordNameStrategy prefixes that with the topic (<topic>-<fqn>), scoping the same record type per topic.

A schema reference lets one schema point to another schema already registered in the registry, so you can reuse a shared type (like an Address) across many schemas instead of copying its definition into each one.

Use ByteArraySerializer when your data is already bytes or you serialize it yourself (e.g. Avro/Protobuf done manually). Use StringSerializer for text. Use typed serializers (Long/Integer/UUID) when keys/values are those primitives. ByteArray is most flexible but least type-safe.

Set default.deserialization.exception.handler. LogAndContinueExceptionHandler logs the bad record and skips it (offset advances); LogAndFailExceptionHandler logs and stops the stream thread. Choose Continue for best-effort tolerance, Fail when no record may be silently dropped.

KafkaJsonSchemaSerializer serializes a Java object to JSON text, prefixes the magic byte + schema ID, and registers/looks up a JSON Schema (Draft-07) in the registry. Configs like auto.register.schemas, json.fail.invalid.schema (validate against schema), and oneof.for.nullables control schema derivation and validation.

normalize.schemas tells the serializer/registry to canonicalize a schema (sort fields, expand defaults, strip insignificant differences) before registering or looking it up, so logically identical schemas that differ only in formatting map to the same schema ID instead of creating duplicate versions.

POST /subjects/{subject}/versions registers a schema and returns its global ID. GET /schemas/ids/{id} fetches a schema by ID. GET /subjects/{subject}/versions lists versions. POST /compatibility/subjects/{subject}/versions/{version} checks if a new schema is compatible before registering.

Use key.subject.name.strategy for the key schema and value.subject.name.strategy for the value schema. Each takes a fully-qualified strategy class such as TopicNameStrategy (default), RecordNameStrategy, or TopicRecordNameStrategy.

Logical types annotate a primitive Avro type with semantic meaning. decimal is bytes/fixed with precision+scale, timestamp-millis is a long of epoch millis, and uuid is a string. Both sides must share the schema to decode them correctly.