skip to content

Which serializers ship built-in with the Kafka clients library, and what wire format do they produce?

level: middleimportance: must knowfreq 70%

answer

  1. org.apache.kafka.common.serialization
  2. numbers = fixed-width big-endian
  3. String charset via serializer.encoding (UTF-8 default)
  4. UUID = toString() text, 36 bytes
  5. NO built-in JSON/Avro in core jar

basics

~10 s

Kafka bundles StringSerializer, ByteArraySerializer, IntegerSerializer, LongSerializer, DoubleSerializer, ShortSerializer, FloatSerializer, and UUIDSerializer (plus ByteBuffer/Bytes/Void). Numbers use fixed-width big-endian bytes; String/UUID use a configurable charset (UTF-8 default).

solid answer

~40 s

All built-ins live in org.apache.kafka.common.serialization. The common ones are ByteArraySerializer (identity pass-through), StringSerializer (encodes with a charset set by the serializer.encoding config, default UTF-8), IntegerSerializer (4-byte big-endian), LongSerializer (8-byte big-endian), ShortSerializer (2 bytes), FloatSerializer (4 bytes), DoubleSerializer (8 bytes), and UUIDSerializer (the UUID's toString() encoded as text). There's also ByteBufferSerializer, BytesSerializer (for the org.apache.kafka.common.utils.Bytes wrapper), and VoidSerializer. Each has a matching Deserializer. The numeric ones produce fixed-width big-endian (network byte order) representations — important to know if a non-Java consumer reads the topic. Notably there is NO built-in JSON or Avro serializer in the core clients jar; JSON support comes from kafka-streams (JsonSerializer/connect) or third parties, and Avro/Protobuf via Confluent's Schema Registry serdes.

go deeper

for a junior

Recognize the names: String, ByteArray, Integer, Long, Double, UUID serializers exist out of the box.

for a middle

Know the package, that numbers are fixed-width big-endian, and that JSON/Avro are not in core.

for a senior

Reason about cross-language interop: byte widths, endianness, charset config, UUID-as-text.

for a principal

Advise on wire-format contracts across polyglot consumers and why teams standardize on a schema serde over raw primitives.

## Where they live Every built-in serializer/deserializer sits in the package `org.apache.kafka.common.serialization`, shipped inside the `kafka-clients` jar. Each type comes as a matched pair, e.g. `LongSerializer` + `LongDeserializer`. ## The catalog - **ByteArraySerializer / ByteArrayDeserializer** — identity. The value already *is* a `byte[]`; serialize returns it unchanged. Use when you do your own encoding. - **StringSerializer / StringDeserializer** — encode/decode text. The charset is controlled by the `key.serializer.encoding`, `value.serializer.encoding`, or fallback `serializer.encoding` property; **default UTF-8**. - **IntegerSerializer** — 4 bytes, **big-endian** (most significant byte first, i.e. network byte order). - **LongSerializer** — 8 bytes big-endian. - **ShortSerializer** — 2 bytes big-endian. - **FloatSerializer** — 4 bytes (IEEE-754, big-endian bit pattern). - **DoubleSerializer** — 8 bytes (IEEE-754). - **UUIDSerializer** — takes a `java.util.UUID`, calls `toString()` (the 36-char canonical form), then encodes that text with the configured charset. It is **text**, not the 16 raw bytes. - **ByteBufferSerializer** — serializes a `java.nio.ByteBuffer`. - **BytesSerializer** — for Kafka's `org.apache.kafka.common.utils.Bytes` immutable wrapper. - **VoidSerializer** — always produces `null` bytes; used when a key or value is conceptually absent. ## Wire format details that bite people - Numeric serializers are **fixed-width and big-endian**. A `LongSerializer` value is always 8 bytes, even for the number 5. A consumer in Python/Go must read 8 bytes big-endian, not parse text. - `StringSerializer` writes the raw encoded bytes with **no length prefix and no terminator** — the message length is the record's byte length. - `UUIDSerializer` is **textual**, so a UUID occupies 36 bytes, not 16. People often assume the compact binary form. ## What is NOT built in There is **no JSON serializer and no Avro serializer in the core `kafka-clients` jar**. JSON serdes ship with Kafka Streams (`org.apache.kafka.connect.json` / Streams' JSON Serde) or come from libraries like Spring Kafka's `JsonSerializer`. Avro/Protobuf/JSON-Schema serdes come from Confluent's Schema Registry packages. Knowing this prevents the misconception that Kafka 'natively serializes JSON.' ## Null handling All built-ins return `null` from `serialize()` when given `null` input (and produce/consume tombstones cleanly). See the null/tombstone question for why that matters in compacted topics.

  • How many bytes does LongSerializer produce for the value 1?
    Always 8 bytes, big-endian: 00 00 00 00 00 00 00 01. It is fixed-width, not a variable-length or text encoding.
  • Does Kafka have a built-in JSON serializer in kafka-clients?
    No. JSON serdes come from Kafka Streams, Spring Kafka, or third parties; the core clients jar only ships primitive/byte/String/UUID serdes.

saying these in an interview costs you the question

  • Claiming numeric serializers use little-endian or text/varint encoding — they are fixed-width big-endian.
  • Saying UUIDSerializer writes the 16 raw bytes — it writes the 36-char string form.
  • Asserting kafka-clients ships a JSON or Avro serializer.

context

resolving links…