Monitoring and Performance Tuning

UnderReplicatedPartitions counts how many partitions led by this broker have fewer in-sync replicas than the configured replication factor. In a healthy cluster it should be 0. A sustained non-zero value means replicas are falling behind or down.

JMX (Java Management Extensions) is the Java standard for exposing runtime metrics as MBeans. A Kafka broker publishes its internal metrics as JMX MBeans. You enable it by setting JMX_PORT (or jmxremote system properties) and read it with tools like JConsole, jmxterm, or a Prometheus JMX exporter.

num.network.threads handle reading requests off and writing responses onto network sockets; num.io.threads do the actual work (reading/writing the log on disk). Default 3 and 8. Raise them if request-handler/network idle time drops.

Kafka clients expose metrics over JMX. Key producer metrics: record-send-rate, request-latency-avg, buffer-available-bytes. Key consumer metrics: records-consumed-rate, fetch-latency-avg, records-lag-max. They show throughput, latency, and buffering/lag health.

max.poll.records caps how many records one poll() call returns. If processing each batch takes too long, the consumer misses its deadline and gets removed from the group. Lowering it means smaller batches that process faster, keeping the consumer alive.

Consumer lag is how far behind a consumer is on a partition: log-end offset minus the consumer's committed offset. A lag of 0 means the consumer has read everything; a growing lag means it can't keep up with producers.

batch.size is the max bytes per partition batch; linger.ms is how long the producer waits to fill a batch before sending. Raising either groups more records per request, boosting throughput but adding latency.

TotalTimeMs, exposed via JMX under kafka.network:type=RequestMetrics, name=TotalTimeMs, request=<ApiKey> (like Produce or Fetch). It's a histogram, so you read Mean, 99thPercentile, and Max.

They are JMX rate meters on the broker: BytesInPerSec is bytes produced into the broker per second, BytesOutPerSec is bytes consumers fetch out per second, and MessagesInPerSec is records produced per second. They exist both per-broker and per-topic.

ActiveControllerCount is per-broker and is 1 on the single controller broker and 0 on the rest, so cluster-wide it must sum to exactly 1. OfflinePartitionsCount is the number of partitions with no leader — healthy is 0; any positive value means those partitions are unavailable.

Kafka writes to the OS page cache and lets the OS flush to disk in the background, which is fast. Durability comes from replication, not fsync. log.flush.interval.* would force periodic fsyncs, but Kafka leaves them effectively off by default.

fetch.min.bytes tells the broker not to answer a fetch until it has at least that many bytes ready. fetch.max.wait.ms caps how long it waits for that. Bigger min.bytes = fewer, larger fetches (more throughput, more latency); the wait prevents waiting forever when traffic is low.

kafka-consumer-groups --describe shows, per partition, CURRENT-OFFSET (committed), LOG-END-OFFSET, LAG (the difference), plus consumer-id/host. Offset lag counts messages behind. End-to-end latency is the time between when a record was produced and when it is consumed — a time, not a count — so high-throughput partitions can have big offset lag but tiny time latency.

records-lag is a per-partition consumer client (JMX) metric showing how far behind that fetch is; records-lag-max is the maximum across the consumer's assigned partitions. They come from the live client, so they only exist while the consumer is fetching and reflect what it has fetched, not committed.

acks controls how many brokers must confirm a write. acks=0 (no wait) is fastest but can lose data; acks=1 waits for the leader; acks=all waits for all in-sync replicas — most durable but highest latency.

Kafka splits each request's time into phases: time waiting in the request queue, time processing locally, time waiting on other brokers, time in the response queue, and time sending the response. TotalTimeMs is their sum.

Idle percent is the fraction of time a thread pool sits with no work. NetworkProcessorAvgIdlePercent and RequestHandlerAvgIdlePercent near 0 mean the network or I/O threads are saturated, which causes requests to queue up and tail latency to spike.

Kafka breaks each request's total time into phases: queue, local processing, remote (waiting on other brokers), throttle, and response send. You read these phase metrics (RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs) to see which phase dominates.

Disk needed = ingress bytes/sec × retention seconds × replication factor, summed across topics, plus headroom. Use BytesInPerSec for the rate and retention.ms (or retention.bytes) for how long data is kept.

Run kafka-producer-perf-test.sh repeatedly at increasing fixed --throughput values and record p99 latency each time. Plot offered rate (x) against latency (y). Latency stays flat then sharply rises at the 'knee' — the point where you hit saturation.

Within a consumer group, each partition is consumed by exactly one member. So the number of partitions is the maximum useful parallelism. Once you have as many consumers as partitions, extra consumers sit idle and throughput stops rising.

With acks=all a produce request can't complete until all in-sync replicas (the ISR) catch up. A slow follower lags, so the produce waits longer (high RemoteTimeMs); if it lags past replica.lag.time.max.ms the leader drops it from ISR (ISR shrink). Correlate RemoteTimeMs, replica lag, ISR-shrink rate, and the follower's own health.

It runs a consumer that reads a fixed number of messages from a topic and reports how much data and how many records it consumed per second. You read its MB/sec and nMsg/sec columns to judge consume throughput.

MB/s is data-volume throughput (bytes per second); records/s is message-count throughput (messages per second). They differ because record size links them: small records can give high records/s but low MB/s, and vice versa. 'Sustained' means the rate held over a long, steady run, not a peak burst.

Early in a run the JVM JIT, OS page cache, TCP connections, and producer batching haven't stabilized, so the first numbers are misleadingly slow. Run long enough and discard the warm-up interval, quoting only the steady-state rate.

Bind Kafka's client metrics into Micrometer using KafkaClientMetrics (or Spring Boot's auto-binding for KafkaTemplate/listener containers). Micrometer then exports them to your backend (Prometheus, OTel). It reads the client's metrics() map and registers them as Micrometer gauges.

MetricsReporter is a pluggable interface (org.apache.kafka.common.metrics.MetricsReporter). You implement it, register it via the metric.reporters client config, and Kafka calls your code as metrics are created/changed/removed so you can forward them (e.g. to Prometheus or a custom sink).

fetch.max.bytes caps the total size of a whole fetch response across all partitions; max.partition.fetch.bytes caps how much comes from each single partition. Historically, if a message was bigger than max.partition.fetch.bytes the consumer could stall — modern clients still return that oversized record so the consumer can progress.

compression.type compresses each batch before sending, cutting network and disk usage and raising throughput. snappy/lz4 are fast with modest ratios; zstd compresses harder (smaller payloads) at more CPU; gzip is highest ratio but slowest.