How do num.io.threads and num.network.threads affect broker throughput, and how would you decide whether to increase one of them?

Kafka's two tunable broker thread pools map directly onto the two halves of the request pipeline, and tuning them is about matching pool size to where work actually queues. **`num.network.threads` (default 3) — Processor pool.** These threads do **only network I/O**: read bytes off sockets, parse into requests, enqueue on the shared request queue, and later write responses. They are non-blocking NIO event loops. They become the bottleneck when there is a lot of *connection/byte* work: many clients, high request rate, TLS encryption/decryption (which happens here), or large response serialization. **`num.io.threads` (default 8) — request-handler (I/O) pool.** These threads dequeue requests and run **`KafkaApis`** logic: appending to the log, reading log segments (page-cache or disk), driving replication, handling metadata. They become the bottleneck when there is a lot of *work per request*: heavy produce/fetch volume, slow disks, compression/decompression, or many partitions. **How to decide — the idle-percent metrics.** Each pool exposes an average idle fraction in `[0,1]`: - **`RequestHandlerAvgIdlePercent`** (kafka.server:type=KafkaRequestHandlerPool) — fraction of time handler threads sit idle waiting for work. - **`NetworkProcessorAvgIdlePercent`** (kafka.network:type=SocketServer) — same for processors. A value near **0.0** means that pool is saturated (the bottleneck); near **1.0** means it has spare capacity. Rule of thumb: keep idle above ~0.3; if it persistently approaches 0, add threads to *that* pool. **Corroborating signals before you change anything:** - **`RequestQueueSize`** climbing toward `queued.max.requests` (500) plus high **`RequestQueueTimeMs`** means requests are waiting for a handler → raise `num.io.threads`. - High **`LocalTimeMs`** (time the handler spends doing the work itself) with low queue time means the bottleneck is downstream (disk/replication), and more threads won't help — fix I/O or partitioning instead. - Low processor idle + high `ResponseQueueTimeMs` points at the network pool → raise `num.network.threads`. **Sizing heuristics.** `num.io.threads` is often raised toward the number of CPU cores or disks (commonly 8–16, higher on big NVMe boxes). `num.network.threads` is raised when many connections/TLS dominate (e.g. 3 → 6–8). Both can be changed dynamically as cluster-wide dynamic configs without a restart in modern Kafka. **Cautions.** Over-provisioning increases context switching, lock contention on shared structures, and memory; it can *reduce* throughput. Always tune one pool at a time, driven by the idle metric for that pool, and re-measure. Adding handler threads cannot overcome a disk- or network-bound broker — those need hardware, better partitioning, or producer batching changes.