How would you implement a custom Serializer/Deserializer, and what production concerns must it handle?
answer
- serialize/deserialize + configure(isKey) + close
- null in -> null out (tombstone-safe)
- stateless / thread-safe (ObjectMapper ok, SimpleDateFormat not)
- embed a version byte for evolution
- throw SerializationException, avoid poison-pill stall
- hot path -> minimize allocation
basics
~20 sImplement Serializer<T>/Deserializer<T>, putting your encoding in serialize()/deserialize(). Handle null in and out, make it thread-safe and stateless, honor the isKey flag in configure(), version your format for evolution, and fail with SerializationException — never crash the whole consumer.
solid answer
~50 sA custom serializer implements org.apache.kafka.common.serialization.Serializer<T>: serialize(topic, data) returns byte[], with optional configure(Map<String,?>, boolean isKey) and close(). The deserializer mirrors it. Production concerns: (1) Null handling — return null for null input and tolerate null bytes on read. (2) Thread safety — one serializer instance is shared across the producer's send path; keep it stateless or use thread-safe encoders (a Jackson ObjectMapper is thread-safe; a raw SimpleDateFormat is not). (3) The isKey flag lets one class behave differently for keys vs values and read key-/value-specific configs. (4) Versioning — embed a format/schema version byte so the deserializer can evolve; otherwise old consumers break on new bytes. (5) Robust failure — throw SerializationException (not arbitrary runtime exceptions) so Streams' exception handlers or your consumer's error logic can react rather than poison the partition. (6) Avoid heavy per-record allocation; serialization is on the hot path. Schema Registry's magic-byte framing is a sibling concern, not covered here.
go deeper
Know the interfaces exist and that you put encoding logic in serialize/deserialize.
Implement one correctly, handle null, and use SerializationException for bad input.
Address thread-safety, isKey, versioning, poison pills, and hot-path performance.
Set org-wide serde standards: format evolution policy, error-handling strategy, dead-lettering, and when to mandate a schema registry instead of bespoke formats.
## The two interfaces ``` public interface Serializer<T> { default void configure(Map<String,?> configs, boolean isKey) {} byte[] serialize(String topic, T data); default byte[] serialize(String topic, Headers headers, T data) { return serialize(topic, data); } default void close() {} } public interface Deserializer<T> { default void configure(Map<String,?> configs, boolean isKey) {} T deserialize(String topic, byte[] data); default T deserialize(String topic, Headers headers, byte[] data) { return deserialize(topic, data); } default void close() {} } ``` The producer/consumer instantiate your class by reflection (from `key.serializer`/`value.serializer` etc.) and call `configure` once, then `serialize`/`deserialize` per record. ## Concern 1 — null contract By convention `serialize(topic, null)` returns `null`, and `deserialize(topic, null)` returns `null`. Breaking this turns harmless tombstones into NPEs or, worse, into bogus non-null payloads. Always branch on null first. ## Concern 2 — thread safety and statelessness A producer creates **one** serializer instance and uses it from the sending thread; consumers/Streams reuse one deserializer per thread but the same instance may be shared. Keep instances **stateless**. If you must hold an encoder, ensure it's thread-safe: Jackson's `ObjectMapper` is safe to share for read/write; `SimpleDateFormat` and many mutable buffers are **not**. Mutable per-instance state causes data corruption under load. ## Concern 3 — the isKey flag `configure(configs, isKey)` receives `isKey=true` when the instance is wired as the key (de)serializer and `false` for the value. This lets a single class read different properties (e.g. `my.key.charset` vs `my.value.charset`) or encode keys and values differently. ## Concern 4 — format versioning / evolution Raw custom formats are brittle: if you add a field, old deserializers misread new bytes. Embed a leading **version byte** (or a self-describing schema) so the deserializer can dispatch on it. This is the home-grown analog of what Schema Registry formalizes — but Schema-Registry magic-byte framing is owned by a sibling topic and out of scope here. ## Concern 5 — error handling discipline Throw `org.apache.kafka.common.errors.SerializationException` on bad input. In a consumer, an uncaught exception from `deserialize` can repeatedly fail the same offset (a 'poison pill'), stalling the partition. In Streams, throwing the right type lets `default.deserialization.exception.handler` (LogAndContinue / LogAndFail) decide whether to skip or stop. Swallowing errors and returning corrupt objects is worse than failing loudly. ## Concern 6 — performance Serialization runs **per record on the hot path** (synchronously inside `send()`/poll processing). Avoid per-call allocation of expensive objects, reuse buffers carefully (only if thread-safe), and prefer compact encodings. A slow serializer directly inflates producer latency and reduces throughput. ## Lifecycle `configure` is called once at startup; `close` once at shutdown to release resources. Don't open per-record connections in serialize/deserialize.
- A consumer keeps crashing on one offset and never advances. What is happening and how do you handle it?A poison pill: a record whose deserializer throws repeatedly at the same offset. Fix the deserializer to throw SerializationException and use an error-handling deserializer/handler (e.g. Streams LogAndContinue, or skip/dead-letter the bad record) so the partition isn't stalled.
- Why does the configure method receive an isKey boolean?So one Serializer class can be reused for both key and value, reading key- vs value-specific configuration or applying different encoding depending on which side it serves.
saying these in an interview costs you the question
- Holding mutable, non-thread-safe state (e.g. SimpleDateFormat) in the serializer.
- Letting serialize throw NPE on null instead of returning null.
- No format versioning, so any field change breaks existing consumers.
- Throwing generic RuntimeExceptions that bypass Streams/consumer error handling.
- Doing heavy per-record allocation or I/O on the serialization hot path.