skip to content

A consumer is being evicted because each batch takes too long to process. How does reducing max.poll.records help, and what are the tradeoffs?

level: middleimportance: should knowfreq 58%

answer

  1. default 500 records per poll
  2. client-side cap, applied after fetch
  3. doesn't change network bytes (fetch.max.bytes)
  4. smaller batch → faster loop → more headroom
  5. no help if ONE record is slow

basics

~20 s

max.poll.records caps how many records each poll() returns. Lowering it means smaller batches that process faster, so poll() is called again sooner and you stay under max.poll.interval.ms. The tradeoff is more poll round-trips and potentially lower throughput.

solid answer

~40 s

max.poll.records (default 500) limits the number of records a single poll() returns; it's a client-side cap applied after fetching, so it doesn't change network fetch sizes (those are governed by fetch.max.bytes / max.partition.fetch.bytes). If processing is roughly linear in record count, halving max.poll.records halves per-batch processing time, giving more headroom against max.poll.interval.ms. It's the cheapest, safest lever because it doesn't touch group-stability timers. Tradeoffs: more frequent poll() calls add overhead and can reduce throughput if you lose batching efficiency downstream (e.g. fewer records per DB bulk insert); and it only helps if processing time scales with count — a single pathologically slow record (one huge message, one slow external call) still blows the interval regardless of batch size. In that case you need pause()/resume() or async offload instead.

go deeper

for a junior

Know max.poll.records limits records per poll (default 500) and that lowering it makes each loop faster.

for a middle

Explain it's a client-side cap (not network bytes), why smaller batches give max.poll.interval.ms headroom, and the throughput/batching tradeoff.

for a senior

Reason about linear vs. non-linear processing cost and when this lever fails, plus its relation to fetch.max.bytes/max.partition.fetch.bytes.

for a principal

Choose between this and pause/resume/offload based on cost model, and balance loop overhead, commit granularity, and downstream batch efficiency.

## What max.poll.records actually does `max.poll.records` (default **500**) caps how many records a single `poll()` returns to your application. It is purely a **client-side** limit applied *after* the consumer has fetched data from the broker. The bytes fetched over the network are controlled by other configs: - `fetch.min.bytes` / `fetch.max.bytes` — broker-side fetch sizing for the whole request. - `max.partition.fetch.bytes` — per-partition cap. The consumer keeps an internal buffer of fetched records and dispenses at most `max.poll.records` per `poll()` from it. So lowering `max.poll.records` does NOT reduce network traffic; it reduces the *unit of work per loop iteration*. ## Why it relieves max.poll.interval.ms pressure The time between two `poll()` calls is dominated by processing the returned batch. If processing cost is approximately linear in the number of records: ``` time_between_polls ≈ max.poll.records × per_record_cost ``` Cut `max.poll.records` from 500 to 100 and per-batch time drops ~5×, giving you a large margin below `max.poll.interval.ms`. Because it doesn't touch `session.timeout.ms` or `heartbeat.interval.ms`, it can't destabilize group membership the way mis-tuning those can. That makes it the **first lever to try**. ## Tradeoffs and limits 1. **More overhead.** More poll() iterations means more coordinator round-trips, more commit calls (if you commit per batch), and less amortization of fixed per-batch costs. Throughput can dip. 2. **Downstream batching loss.** If you bulk-write to a DB or another topic per batch, smaller batches mean smaller bulk operations and possibly worse downstream efficiency. 3. **Doesn't help non-linear cases.** If one record triggers a 60-second external call, shrinking the batch to a single record still takes 60 seconds. Smaller batches only help when total time scales with count. 4. **Commit granularity.** Smaller batches commit progress more often, which reduces reprocessing on rebalance — a side benefit. ## Decision guide - Processing time scales with count, occasionally spikes → **lower max.poll.records** (cheap, safe). - Individual records are slow/variable → **pause()/resume() + offload to a worker pool**, and/or raise `max.poll.interval.ms`. - Legitimately long but bounded processing → **raise max.poll.interval.ms** to exceed worst-case.

  • Does lowering max.poll.records reduce the amount of data fetched from the broker over the network?
    No. It's a client-side cap on how many buffered records each poll() hands back. Network fetch sizing is governed by fetch.max.bytes and max.partition.fetch.bytes; the consumer just dispenses fewer records per poll from its buffer.
  • When does lowering max.poll.records NOT solve a max.poll.interval.ms eviction?
    When processing time doesn't scale with record count — e.g. a single oversized message or one slow synchronous external call. Then even a batch of one record exceeds the interval, and you need pause()/resume() or async offload.

saying these in an interview costs you the question

  • Claiming max.poll.records reduces network fetch size — it's a post-fetch client-side cap.
  • Assuming smaller batches always fix the eviction — useless when a single record is slow.
  • Confusing max.poll.records with max.partition.fetch.bytes (count vs. bytes).

context

resolving links…