Spring for Apache Kafka

DefaultErrorHandler is the container's default handler when a @KafkaListener throws. It re-delivers the failed record several times (using a BackOff for delays), and if all attempts fail it hands the record to a recoverer.

@KafkaListener marks a method to consume messages from a Kafka topic. @EnableKafka turns on Spring's infrastructure that finds those methods and starts listener containers to feed them records.

Put @EnableKafkaStreams on a @Configuration class and provide a KafkaStreamsConfiguration bean named defaultKafkaStreamsConfig that sets the application.id and bootstrap.servers. Spring then creates a StreamsBuilderFactoryBean that builds and starts Kafka Streams.

KafkaTemplate is Spring's helper for producing messages. You inject it and call template.send("topic", key, value). Spring Boot auto-configures it from application.properties, so you just autowire and send.

read_committed makes the consumer skip records from aborted transactions and not read records from transactions that are still open. It only sees data the producer actually committed, so a rolled-back send never becomes visible.

It's a recoverer that, after retries are exhausted, republishes the failed record to a dead-letter topic (default: original topic name + ".DLT") using a KafkaTemplate, adding headers describing the failure so the message isn't lost.

Concurrency is how many consumer threads (each its own KafkaConsumer) the container runs. Kafka gives each partition to at most one consumer in a group, so useful concurrency is capped by partition count — extra threads stay idle.

Write a @Bean method that takes a StreamsBuilder parameter. Spring injects the default StreamsBuilder, you build your KStream/KTable pipeline on it, and the StreamsBuilderFactoryBean builds the topology and auto-starts the KafkaStreams instance after the context refreshes.

send() is asynchronous — it returns a future that completes when Kafka acknowledges the write. You attach whenComplete/thenAccept callbacks to handle the SendResult on success or the exception on failure, instead of assuming send succeeded.

Give the producer a transactional.id (in Spring, a transactionIdPrefix on the producer factory) so it can run transactions and be fenced across restarts. Then use KafkaTransactionManager so KafkaTemplate sends inside @Transactional commit or roll back atomically.

@RetryableTopic makes retries non-blocking: a failed record is forwarded to separate retry topics (with delays), so the main consumer keeps processing new records instead of being stuck. After the configured attempts, it goes to a DLT. DefaultErrorHandler instead retries in place by seeking, which blocks the partition.

AckMode controls when the container commits offsets. RECORD commits after each record; BATCH commits after the whole poll batch (default); MANUAL/MANUAL_IMMEDIATE hand you an Acknowledgment object so your code decides when to commit.

Configure the listener container with a KafkaTransactionManager. For each batch it begins a transaction, runs your listener's sends, then commits the consumed offsets into that same transaction via the producer, and commits once. If anything fails it aborts, so neither the output nor the offset advance.

Use TopicBuilder.name("t").partitions(n).replicas(m), then add topic configs via .config(key, value) or .compact()/.configs(map). .build() returns a NewTopic bean that KafkaAdmin provisions.

Configure the value-serializer as Spring's JsonSerializer. It uses Jackson to turn your POJO into JSON bytes. Then send(topic, key, pojo) works directly. The consumer side uses JsonDeserializer to reconstruct the object.

By default it's non-fatal: KafkaAdmin logs that it couldn't reach the broker and the app still starts, but the declared topics are not created. Set fatalIfBrokerNotAvailable (spring.kafka.admin.fail-fast) to make startup fail instead.

It won't recreate the topic. It can increase partitions to match a higher declared count (never decrease). It ignores replication-factor differences. Topic configs are only altered if modifyTopicConfigs is enabled; otherwise differences are left alone.

It uses an exception classifier. Some exceptions (like deserialization or conversion errors) are fatal by default — retrying can't help — so they skip retries and go straight to the recoverer. You adjust the lists with addRetryableExceptions / addNotRetryableExceptions.

A batch listener receives a whole List of records from one poll() in a single method call instead of one record at a time. You enable it on the container factory (setBatchListener(true)) and the method takes a List; with manual ack, one Acknowledgment covers the entire batch.

Set a StreamsUncaughtExceptionHandler on the StreamsBuilderFactoryBean to decide whether to replace the thread, shut down the client, or shut down the whole app. For anything you can't set via properties, register a KafkaStreamsCustomizer that receives the KafkaStreams before start().

StreamsBuilderFactoryBean is a Spring FactoryBean that produces the StreamsBuilder and also manages the KafkaStreams lifecycle. It is configured from a KafkaStreamsConfiguration (the StreamsConfig properties) and, as a lifecycle bean, builds the topology and starts/stops Kafka Streams with the context.

The key decides the partition: Kafka hashes the key and maps it to a partition, so all records with the same key go to the same partition and stay ordered. With a null key, records are spread across partitions. A custom Partitioner overrides this mapping.

ProducerFactory creates the underlying Kafka Producer objects for KafkaTemplate, holding the config and serializers. DefaultKafkaProducerFactory is the standard implementation; by default it creates one shared, thread-safe producer that all sends reuse.

Use executeInTransaction for a quick local, producer-only transaction without any transaction manager or Spring context. Use @Transactional with KafkaTransactionManager when you want declarative transactions that can coordinate with the listener container or other resources.

Match the mechanism to the constraint: use blocking seek retries (DefaultErrorHandler) where per-partition ordering matters and outages are short; use non-blocking @RetryableTopic where throughput and long back-offs matter and ordering can be relaxed. Always classify transient vs fatal, and route the unrecoverable to a DLT with replay tooling.

ContainerProperties is the configuration object on a listener container holding container-level settings: ack mode, poll timeout, ack time/count, consumer rebalance listener, sync/async commits, and the task executor. You tune it for commit behavior and rebalance stability.

Set acks=all and enable.idempotence=true on the ProducerFactory for durable, non-duplicating writes. For atomic multi-message sends, add a transactionIdPrefix so the template runs Kafka transactions. The trade-off is lower throughput and higher latency for stronger guarantees.

Kafka EOS guarantees exactly-once only for read-process-write within Kafka: idempotent producers plus transactions make produce+offset-commit atomic. It does not cover external systems like databases or HTTP calls, so those must be idempotent. Fencing and read_committed complete the picture.

It's great for local/dev and convenience, but as the sole production strategy it's weak: it won't change replication factor, tolerates config drift by default, can silently create 1/1 topics, and ties correct topic layout to app startup. Prefer infra-as-code for prod and use KafkaAdmin declaratively.