skip to content

You need to maximize producer throughput for a high-volume Kafka pipeline. Which producer configs do you tune together, and what are the trade-offs and failure modes?

level: principalimportance: should knowfreq 40%

answer

  1. batch.size + linger.ms = core lever (per-partition!)
  2. compression lz4/zstd synergizes with big batches
  3. buffer.memory + max.block.ms = back-pressure / TimeoutException
  4. acks/idempotence/in-flight<=5 ordering
  5. sticky partitioner (KIP-480) fills batches
  6. benchmark throughput AND p99

basics

~20 s

Raise batch.size and linger.ms so batches fill, enable a fast codec like lz4 or zstd via compression.type, and increase buffer.memory so the producer doesn't block. Accept added per-record latency and watch for buffer exhaustion, ordering, and broker size limits.

solid answer

~40 s

For raw throughput I tune four knobs together and accept higher latency. Raise batch.size (e.g. 64-256 KB) and linger.ms (e.g. 10-100 ms) so batches actually fill — these two are the core lever. Enable compression.type=lz4 or zstd, which both shrinks network/disk and, because compression is per-batch, rewards the bigger batches. Increase buffer.memory (default 32 MB) so producers don't block on a full accumulator under bursts, and tune max.block.ms for the back-pressure policy. Trade-offs and failure modes: bigger linger adds tail latency; if buffer.memory is too small, send() blocks then throws TimeoutException; oversized batches can hit max.request.size or the broker's message.max.bytes (RecordTooLargeException); with acks=all and idempotence, max.in.flight.requests.per.connection interacts with ordering and retries. I also size batch.size relative to partition count (each partition has its own batch) and validate end-to-end with throughput/latency benchmarks before committing.

go deeper

for a junior

Know that bigger batch.size/linger.ms plus compression raise throughput at a latency cost.

for a middle

Tune the four knobs together and name the back-pressure failure (TimeoutException from buffer.memory).

for a senior

Reason about per-partition memory, sticky partitioner, idempotence/in-flight/ordering, and size-limit rejections.

for a principal

Drive a benchmark-led, SLA-aware tuning method; standardize defaults; weigh durability/ordering/cost trade-offs across the org and choose claim-check for large payloads.

## The throughput tuning toolkit Maximizing producer throughput is a coordinated tuning exercise, not a single switch. The interacting configs: ### Core lever: batch.size + linger.ms - **batch.size** (default 16 KB): raise to 64-256 KB so each request carries more records. Remember it's **per partition** — total memory pressure scales with active partitions. - **linger.ms** (default 0): raise to ~5-100 ms so batches actually fill before sending. This is what turns a low/medium-rate stream into full batches. Adds per-record latency equal to (up to) linger.ms. Together they decide how full your batches get. Full batches = fewer requests = less per-request overhead and broker CPU = higher throughput. ### Compression: compression.type Enable **lz4** or **zstd**. Because compression is per-batch, larger batches compress better, so this synergizes with the lever above. Compression cuts network egress, replication traffic, and disk — often the real throughput ceiling. Cost is producer CPU; lz4/zstd keep that low. zstd gives the best ratio for bandwidth/storage-bound pipelines. ### Memory / back-pressure: buffer.memory + max.block.ms - **buffer.memory** (default 32 MB): total memory for unsent batches. If producers outrun brokers, this fills; `send()` then blocks up to **max.block.ms** (default 60 s) and throws **TimeoutException** if it can't get space. Raise buffer.memory for bursty workloads; tune max.block.ms to choose fail-fast vs wait. ### Delivery semantics that interact - **acks**: acks=all (with min.insync.replicas) is safest but slower; acks=1 trades durability for latency. Modern producers default to acks=all with idempotence. - **enable.idempotence=true** (default in recent Kafka) caps **max.in.flight.requests.per.connection** at 5 and preserves ordering with retries. Don't set in-flight >5 with idempotence, and don't disable ordering guarantees for throughput without understanding the risk. - **retries / delivery.timeout.ms**: govern resilience; high in-flight + retries without idempotence can reorder. ## Failure modes to anticipate 1. **Latency regression**: linger.ms directly adds to p50/p99 produce latency. Set it from your latency SLA, not arbitrarily. 2. **Buffer exhaustion**: too-small buffer.memory under bursts → send() blocks → TimeoutException → dropped or back-pressured upstream. Monitor `buffer-available-bytes` and `record-queue-time`. 3. **Size-limit rejections**: large batches can exceed max.request.size (producer) or message.max.bytes (broker) → RecordTooLargeException. Keep these aligned (see message-size question). 4. **Partition skew**: batch.size is per-partition, so many partitions multiply memory and may under-fill batches if keys spread thin; sticky partitioning (KIP-480) helps fill batches. 5. **Ordering surprises**: tuning in-flight requests up without idempotence can reorder on retry. ## Method, not magic numbers - Start from the workload: record size, rate, partition count, latency SLA, durability requirement. - Tune batch.size/linger.ms first, add compression, then size buffer.memory for burst headroom. - Benchmark with `kafka-producer-perf-test.sh` measuring throughput AND p99 latency. - Watch JMX producer metrics: `batch-size-avg`, `records-per-request-avg`, `compression-rate-avg`, `record-queue-time-avg`, `buffer-available-bytes`. - Iterate; there is no universal config — it's a per-pipeline cost/SLA optimization. ## Sticky partitioner note KIP-480's sticky partitioner (default for null-key records in Kafka 2.4+) sends a burst of records to one partition until a batch fills, which materially improves batch fullness and throughput versus the old round-robin — relevant context when reasoning about why batches fill.

  • What happens if buffer.memory is too small for your throughput?
    When unsent batches fill buffer.memory, send() blocks waiting for space for up to max.block.ms, then throws TimeoutException. This back-pressures or fails the producing application. For bursty high-throughput pipelines, raise buffer.memory and monitor buffer-available-bytes and record-queue-time-avg.
  • How does the sticky partitioner (KIP-480) help throughput?
    For records without keys, the old round-robin partitioner spread records thinly across partitions, under-filling each partition's batch. The sticky partitioner (default in Kafka 2.4+) sends a burst to one partition until its batch fills, then switches — producing fuller batches, fewer requests, and better throughput and compression.
  • Why might raising max.in.flight.requests.per.connection risk reordering?
    Without idempotence, if request N fails and is retried while request N+1 already succeeded, records can be reordered. enable.idempotence=true (default) prevents this and caps in-flight at 5, preserving order across retries. So you can't naively crank in-flight for throughput without keeping idempotence on.

saying these in an interview costs you the question

  • Treating batch.size as global rather than per-partition memory
  • Cranking linger.ms without regard to the latency SLA
  • Ignoring buffer.memory exhaustion and the resulting TimeoutException
  • Raising in-flight requests for throughput while breaking ordering guarantees
  • Picking magic-number configs without benchmarking p99 latency

context

resolving links…