How should you size the JVM heap and choose a garbage collector for a Kafka broker, and why is a relatively small heap usually correct?

## Why the heap is small A common surprise: Kafka brokers run very large data volumes on a **small** JVM heap. The reason is architectural — the actual message data is stored in **log segment files** and cached by the **OS page cache** (kernel RAM), entirely **off** the JVM heap. The heap is only used for: - network request/response buffers and the request queue, - per-connection and per-partition metadata and replication/fetch state, - index structures, controller/metadata in KRaft, and assorted bookkeeping. None of that scales with total data stored; it scales with **partitions, connections, and request volume**. So a modest heap suffices regardless of how many TB the broker holds. ## Recommended sizing - **Heap size:** a fixed, modest value — Confluent's guidance is roughly **5–6 GB** for typical brokers; very large/partition-heavy brokers may go higher, but doubling the heap is rarely the right fix. - **Set `-Xms == -Xmx`** so the JVM doesn't resize the heap at runtime (resizing causes pauses and fragmentation). - **Leave the rest of RAM for page cache.** This is the crucial point: if a box has 64 GB, you give the broker ~6 GB heap and let ~50+ GB serve as page cache. Page-cache RAM is what keeps reads off disk and powers zero-copy; spending it on an oversized heap *hurts* performance and lengthens GC. ## Garbage collector - **G1GC (Garbage-First)** is the recommended and default collector for Kafka on modern JDKs (Kafka's startup scripts configure G1). G1 is a low-pause, region-based collector that tries to meet a **pause-time target** rather than minimizing total GC time. - Kafka's tuned defaults include **`-XX:MaxGCPauseMillis=20`** (aim for ~20 ms pauses) and `-XX:InitiatingHeapOccupancyPercent=35` (start concurrent marking earlier). These bias toward short, predictable pauses over raw throughput — correct for a latency-sensitive broker. - On the newest JDKs some operators evaluate **ZGC** for sub-millisecond pauses on large heaps, but G1 remains the standard, well-tested choice for typical broker heaps. ## Why pauses matter operationally A long **stop-the-world** GC pause freezes the broker. Consequences: - The broker can miss its session/heartbeat deadlines and be considered failed, or - fall out of the **ISR** (in-sync replica set) because it stops fetching/acking, triggering leadership churn and replication catch-up. So GC tuning targets *pause predictability*, not throughput. You enable GC logging (`-Xlog:gc*`) and monitor via JMX, alerting on pause spikes and high promotion/allocation rates. ## Diagnosing heap pressure If a broker shows GC pressure or OOMs, the usual root cause is **too many partitions or connections** (each costs heap), not insufficient RAM. The right responses are reducing partition count per broker, capping connections, or scaling out — not blindly enlarging `-Xmx`, which steals page-cache RAM. ## Edge cases / pitfalls - Giving Kafka a huge heap 'to be safe' is an anti-pattern: it starves page cache and produces longer GC pauses — strictly worse. - Swapping is deadly: set `vm.swappiness` low (e.g., 1) so the kernel doesn't swap out broker memory or page cache under pressure. - Heap need grows with partition count; brokers hosting tens of thousands of partitions need more heap and careful GC tuning.