A consumer group has growing lag but CPU on the instances is low and the network isn't saturated. Walk through how you'd diagnose and which fetch/parallelism configs you'd suspect.

## What 'low CPU + rising lag' tells you Lag = (log end offset) - (committed consumer offset). If lag grows while the consumer CPU is idle, the consumers are **not compute-bound** — they're spending time **blocked** (waiting on I/O), **idle** (not enough data per fetch), or **not assigned** the work (skew / under-parallelized). Diagnosis is about finding which. ## Step 1 — Decompose lag per partition Run `kafka-consumer-groups.sh --describe --group <g>` to see per-partition lag. - **One/few partitions lag, rest fine** -> **key skew / hot partition**. A single partition is processed serially by one consumer; if most traffic hashes to one key/partition, that partition bottlenecks no matter how many consumers you add. Fix: better partitioning key or repartition. No fetch config solves this. - **All partitions lag evenly** -> systemic slowness; continue. ## Step 2 — Is the poll loop blocking? Profile or add timing around processing. The classic cause: each record triggers a **synchronous downstream call** (database write, REST call). The poll thread waits on network I/O -> CPU idle but throughput capped by downstream latency. - If so, scale consumers up to partition count, or introduce **in-process parallelism** (worker pool) so multiple records process concurrently. Batch the downstream sink if possible. ## Step 3 — Are you partition-bound? Check instances vs partitions. If instances == partitions, you've hit the parallelism ceiling; adding more does nothing. Options: repartition or in-process parallelism (see Step 2). ## Step 4 — Are fetches starved? If each poll() returns little data: - **max.poll.records** too low -> many poll() calls with per-call overhead. - **fetch.max.bytes / max.partition.fetch.bytes** too small -> small responses, more round trips. - **fetch.min.bytes** high with **fetch.max.wait.ms** high -> consumer waits, throughput on bursty topics can suffer. For catch-up throughput, raise max.poll.records and the fetch byte limits so each round trip moves more data. ## Step 5 — Is it rebalancing? Frequent rebalances stop consumption fleet-wide each time. Causes: - Processing a batch exceeds **max.poll.interval.ms** -> member evicted -> rebalance -> CommitFailedException -> reprocessing -> more lag (a death spiral). Fix: lower max.poll.records or raise the interval; use CooperativeStickyAssignor and static membership to reduce disruption. - Check the **rebalance rate** metric and consumer logs for 'Attempt to heartbeat failed' / 'rebalance'. ## Key metrics to pull - **records-lag-max** (per consumer): max lag across owned partitions. - **fetch-latency-avg/max**: time the broker takes to answer fetches (high = broker-side or network). - **poll-idle-ratio-avg**: fraction of time the consumer is idle inside poll() waiting for data (high = starved fetches or low traffic; low = processing-bound). - **rebalance-rate-per-hour** / **last-rebalance-seconds-ago**: rebalance churn. - **commit-latency**: slow commits can stall the loop. ## Putting it together Low CPU + even lag + high poll-idle-ratio -> fetch starvation or downstream blocking; raise fetch sizes or parallelize. Low CPU + skewed per-partition lag -> repartition/key fix. Frequent rebalances -> poll-interval tuning. The anti-pattern is blindly adding consumer instances when you're already partition-bound or downstream-bound.