Broker Threads, Page Cache and Disk Tuning

## The two thread pools A Kafka broker is a network server that persists data to disk. To keep the network layer and the disk/CPU layer from blocking each other, it splits work across two thread pools connected by a bounded queue. **Network threads — `num.network.threads` (default 3).** Internally these are the `Processor` threads of the `SocketServer`. They run an NIO event loop (a Reactor pattern): one `Acceptor` accepts new TCP connections and round-robins them to processors; each processor reads inbound request bytes off its sockets, assembles complete requests, and pushes them onto a shared **request queue**. After an I/O thread produces a response, the processor writes the response bytes back to the socket. Crucially, network threads do **no** log or disk work — they must never block, or all connections they own stall. **I/O threads — `num.io.threads` (default 8).** These are the `KafkaRequestHandler` threads (the "request handler pool"). Each one loops: take a request from the request queue, dispatch it to `KafkaApis`, perform the real work — appending records to the active log segment, reading bytes for a fetch, serving metadata/offset/group requests — then enqueue the response for the originating network thread. **The queue between them — `queued.max.requests` (default 500).** Bounds how many parsed-but-unhandled requests can pile up. When full, network threads stop reading new requests (backpressure), which protects broker heap from unbounded buffering. ## How to size them — measure, don't guess Kafka exposes idle-percent gauges via JMX: - Network: `kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent` (0.0–1.0). - I/O: `kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent`. Interpretation: idle near 1.0 = pool mostly waiting (fine/over-provisioned); idle trending toward 0 = pool saturated and becoming the bottleneck → add threads. A common operational rule of thumb is to keep both above ~0.3. Increase `num.io.threads` when the request-handler idle drops (often the first bottleneck on disk-heavy workloads); increase `num.network.threads` when the network-processor idle drops (high connection counts / TLS / many small requests). ## Sizing heuristics - I/O threads: scale toward the number of CPU cores and number of data directories (disks); values like 8–16 are common on multi-disk brokers. - Network threads: usually smaller than I/O threads; TLS termination and very high connection counts push it up. - Both are dynamically updatable cluster-wide via `kafka-configs.sh --alter --entity-type brokers` without a restart, which lets you tune from metrics in production. ## Edge cases / pitfalls - Setting them absurdly high doesn't help: extra threads add context-switch overhead and memory; the disk or NIC becomes the real ceiling. - A blocked I/O thread (e.g., slow disk fsync) drains the request queue slowly, request-handler idle drops, and producer/consumer latency spikes — the metric tells you which pool to fix. - Network and I/O idle are independent: a TLS-heavy cluster can saturate network threads while I/O threads sit idle.