skip to content

How does compression.type work in a Kafka producer, and how do gzip, snappy, lz4, and zstd compare?

level: middleimportance: must knowfreq 70%

answer

  1. compress = per BATCH, not per record
  2. gzip=ratio, snappy/lz4=speed, zstd=best balance
  3. zstd via KIP-110 (Kafka 2.1)
  4. codec id in batch header
  5. bigger batch = better ratio

basics

~20 s

compression.type tells the producer to compress each batch as a whole using a codec: gzip (best ratio, slow), snappy (fast, modest ratio), lz4 (fast), or zstd (great ratio and good speed). Compression is per-batch, so bigger batches compress better.

solid answer

~50 s

compression.type (default none) sets the codec the producer applies to each record batch before sending: none, gzip, snappy, lz4, or zstd. Compression happens at the BATCH level — the whole set of records in a batch is compressed together — which is why larger batches (tuned via batch.size/linger.ms) yield much better ratios. Trade-offs: gzip gives the highest compression ratio but is CPU-heavy and slowest; snappy and lz4 are fast with moderate ratios, good for high-throughput low-latency pipelines; zstd (added via KIP-110) typically offers gzip-class ratios at far better speed and is the modern default choice for most workloads. Compression reduces network bytes, broker disk usage, and replication traffic. The codec id is stored in the batch header, so consumers and brokers know how to decompress. Choosing a codec is a CPU-vs-bandwidth-vs-storage decision driven by message shape and throughput.

go deeper

for a junior

Name the codecs and that gzip=small/slow, snappy/lz4=fast, zstd=balanced.

for a middle

Explain per-batch compression, the synergy with batch size, and codec trade-offs by workload.

for a senior

Discuss CPU-vs-bandwidth-vs-storage trade, KIP-110 zstd, codec id in batch header, replication savings.

for a principal

Frame codec choice as a cost-model decision across producer CPU, network egress pricing, disk, and consumer fan-out; standardize defaults org-wide.

## Compression is per-batch, not per-record Kafka compresses an **entire record batch** as one unit, not each record individually. This matters enormously: compression algorithms exploit redundancy across data, and a batch of similar records (same schema, repeated field names in JSON, etc.) compresses far better as a block than each record alone. **Therefore batching and compression are synergistic** — increasing `batch.size`/`linger.ms` directly improves compression ratios. The compressed batch carries a **codec id in its batch header** (the record batch format reserves bits for compression type), so brokers and consumers know how to decompress without extra configuration. ## The codecs `compression.type` accepts: `none` (default), `gzip`, `snappy`, `lz4`, `zstd`. - **gzip** (DEFLATE): highest compression ratio of the classic codecs, but the most CPU-intensive and slowest. Good when bandwidth/storage is the bottleneck and CPU is plentiful. - **snappy** (Google): designed for speed over ratio. Fast compression/decompression, modest ratio. Popular historically for high-throughput pipelines. - **lz4**: very fast, similar niche to snappy, often slightly better. Low CPU overhead. - **zstd** (Facebook/Meta, added to Kafka in **KIP-110**, Kafka 2.1): the modern sweet spot — ratios close to or better than gzip at speeds close to lz4/snappy, with tunable levels. For most new deployments zstd is the recommended choice. ## What compression buys you 1. **Less network egress** from producers and during replication between brokers. 2. **Less disk usage** — Kafka stores batches on disk in their compressed form. 3. **Higher effective throughput** per network request. The cost is **CPU** on the producer (to compress) and on consumers (to decompress). Brokers normally store and serve compressed batches as-is, so they don't pay decompression cost on the hot path (with exceptions — see broker recompression). ## Where it sits in the pipeline The producer compresses after accumulating a batch and before the network send. Consumers decompress on fetch. Because the unit is the batch, a fetch returns compressed batches that the consumer client decompresses in memory. ## Choosing a codec - High-throughput, CPU-constrained producers, latency-sensitive: lz4 or snappy. - Storage/bandwidth-dominated cost, CPU available: gzip or zstd. - General modern default: **zstd** — best overall balance. - Always pair with reasonable batching, or compression has little to chew on. ## Edge case: broker-side codec The topic config `compression.type` can be set to `producer` (keep producer's codec) or a specific codec, which can force the broker to recompress — covered separately.

  • Why do larger batches compress better?
    Compression operates on the whole batch as a block, exploiting redundancy across records (repeated keys, schema fields, similar values). More records per batch means more shared context for the algorithm to deduplicate, raising the compression ratio. This is why batch.size/linger.ms tuning and compression go hand in hand.
  • Which codec would you pick for a high-throughput pipeline where producer CPU is tight, and why?
    lz4 (or snappy) — both are designed for speed with low CPU cost and decent ratios, minimizing producer CPU while still cutting network/disk. If CPU later frees up and storage cost dominates, zstd gives a much better ratio at moderate CPU.

saying these in an interview costs you the question

  • Saying compression is per-record (it's per-batch)
  • Claiming gzip is fastest (it has the best ratio but is slowest)
  • Not knowing zstd exists / when it was added (KIP-110, Kafka 2.1)
  • Thinking brokers always decompress on the hot path (normally they store/serve compressed as-is)

context

resolving links…