Monitoring and Performance Tuning

When a client exceeds its configured quota, the broker delays the request's response and records that delay as ThrottleTimeMs. The delay is also sent back to the client so it can slow itself down.

A hot partition gets a disproportionate share of traffic, so its leader broker works harder and its tail latency rises while cluster averages look fine. Confirm by comparing per-partition/per-broker byte and message rates and checking the producer's partitioning key for skew.

Partition count must be at least target throughput divided by the per-partition throughput a single producer/consumer can sustain. Since one partition maps to one consumer in a group, partitions also set the max parallelism for consumers.

Both are saturation gauges from 0.0 to 1.0. RequestHandlerAvgIdlePercent is the fraction of time the I/O (request handler) threads are idle; NetworkProcessorAvgIdlePercent is the fraction of time the network threads are idle. Low values (near 0) mean those thread pools are saturated and the broker is a bottleneck.

They are rate meters counting how often replicas leave the in-sync replica set (shrink) or rejoin it (expand) per second. Occasional events are normal; a high or oscillating rate means replicas are repeatedly falling behind and catching up — a sign of an overloaded or unstable broker.

Give the broker a modest heap (commonly ~5-6 GB) and leave most RAM to the OS page cache, since Kafka stores data off-heap in files. Use the G1 garbage collector (the default) to keep pauses low.

JBOD gives Kafka each disk separately via log.dirs, so failures are isolated and replication handles redundancy; RAID (often RAID10) adds redundancy and even I/O but costs capacity/write speed. XFS is favored over ext4 for better performance under Kafka's large sequential-write workload.

Zero-copy uses the OS sendfile() call to stream bytes from the page cache straight to the network socket, skipping copies into and out of the JVM heap. It makes consumer fetches cheap — but only for plaintext data Kafka doesn't have to transform.

KIP-714 lets brokers collect standardized client metrics by having clients push them to the broker over the Kafka protocol, instead of operators scraping each client's JMX. The broker subscribes clients to metrics, clients push them as OpenTelemetry-encoded payloads on an interval.

Low CPU + growing lag usually means the consumers are blocked waiting, not working: a slow downstream call, partition skew (one hot partition), too few partitions to parallelize, or fetches that are too small/infrequent. Check per-partition lag, then look at max.poll.records, fetch sizes, and partition assignment.

Burrow watches a sliding window of recent committed offsets per partition and looks at the trend, not one threshold. It flags a group as ERR/STALLED if offsets stop advancing while lag is nonzero, or WARN if lag is consistently growing, instead of alerting on an arbitrary fixed lag number.

kafka-exporter talks the Kafka protocol to the brokers and computes committed lag itself (it works without any live consumer). The JMX exporter scrapes MBeans from a JVM, so it surfaces client metrics like records-lag-max but only while that consumer is running. Use kafka-exporter for durable group lag; JMX exporter for in-process metrics.

buffer.memory is the total bytes the producer can use to buffer unsent records. When it fills (the producer outpaces brokers), send() blocks up to max.block.ms, then throws TimeoutException. It's the producer's backpressure mechanism.

It's the number of unacknowledged produce requests the producer allows per broker connection at once. Higher values pipeline more for throughput; with retries and no idempotence, values above 1 can reorder records on a failed-then-retried batch.

Purgatory is where the broker parks requests that can't complete immediately and must wait for a condition (replication acks, or enough fetch data). That waiting time shows up as RemoteTimeMs.

Requests are piling up in the request queue because there aren't enough I/O (request handler) threads to process them. You typically increase num.io.threads, and check for slow downstream phases that back up the handlers.

Periodic LocalTimeMs spikes point at the leader's local work stalling. Check GC pause logs/JMX for stop-the-world pauses lining up with the spikes; if GC is clean, look at page-cache misses forcing disk reads (rising disk read I/O and read latency while consumers fetch old data).

Kafka has two quota kinds: byte-rate quotas (producer_byte_rate, consumer_byte_rate) that cap MB/s per client, and request-rate quotas (request_percentage) that cap the share of broker request-handler/network thread time. They apply per client-id, user, or user+client-id.

Outbound bandwidth per broker ≈ consumer egress (BytesOutPerSec, multiplied by consumer fan-out) plus replication egress to followers (BytesInPerSec × (replicationFactor − 1)). Inbound ≈ producer ingress plus replica fetch traffic. Replication is often the dominant, easily-forgotten term.

LeaderElectionRateAndTimeMs is a timer the controller exposes: it tracks how often partition leader elections happen and how long they take (in ms). Frequent or slow elections signal instability. For overall broker health, alert on a small curated set — offline/under-replicated partitions, controller count, thread idle, ISR churn, election rate — with thresholds and dwell times tuned to severity.

Low latency: linger.ms=0, small batches, compression none/lz4, acks=1, modest buffer. High throughput: linger.ms=20-100, large batch.size, zstd/lz4 compression, acks=all with pipelining, big buffer.memory. The two sit at opposite ends of the batching/latency curve.

Start by splitting TotalTimeMs into phases to localize the stage, check idle-percent to spot pool saturation, then branch: RemoteTime to followers/ISR, LocalTime to GC/disk/page-cache, queue time to thread pools, send time to network. Compare per-broker to find a hot partition or one bad node.

Trogdor is Kafka's distributed test framework: a coordinator plus agents that run coordinated workloads and fault injections across many nodes. Use it for large-scale, multi-node, repeatable load and chaos tests; use the perf-test scripts for quick single-box benchmarks.

High io-wait-ratio with low send-rate means the producer's network thread is mostly idle waiting for data — it's not the bottleneck. Check record-queue-time, batch-size-avg, request-latency-avg, and buffer-available-bytes to see if the app, batching config, or broker is the limit.