skip to content

What does max.poll.records control, and how does it differ from the fetch-size settings?

level: middleimportance: must knowfreq 70%

answer

  1. max.poll.records = record COUNT, client-side
  2. default 500
  3. fetch.*.bytes = wire BYTES
  4. fetch big, process small
  5. drains buffer, no extra network per slice

basics

~20 s

max.poll.records caps how many records a single poll() call returns to your code (default 500). Fetch-size settings (fetch.max.bytes, max.partition.fetch.bytes) control how much data is pulled from brokers over the wire — bytes, not record count.

solid answer

~40 s

max.poll.records (default 500) is a purely client-side limit on the number of records one poll() returns to your application. It does not change network fetch requests at all — the consumer still fetches large byte-sized chunks from brokers and buffers them, then slices off up to max.poll.records per poll. This decouples the network fetch granularity (governed by fetch.min.bytes, fetch.max.bytes, max.partition.fetch.bytes in bytes) from the application processing granularity (record count). You tune max.poll.records down when per-record processing is heavy, so each poll returns a small, quickly-processed batch and you keep calling poll often enough to stay live and heartbeat. You tune it up for throughput when processing is cheap. Lowering it does not reduce broker load, since fetches are unchanged; it only shapes how records are handed to your loop.

go deeper

for a junior

Know max.poll.records limits how many records one poll returns, default 500.

for a middle

Distinguish record-count cap from byte-size fetch settings and explain the fetch-big/process-small decoupling.

for a senior

Tune it relative to per-record processing cost and poll-loop budget; explain why it doesn't touch network traffic.

for a principal

Reason about end-to-end batch sizing across fetch bytes, buffers, and processing SLAs; set org-wide defaults.

## max.poll.records vs fetch sizing There are two distinct layers and people constantly conflate them. ### Layer 1 — the wire (bytes) When the consumer talks to brokers it issues **Fetch** requests measured in **bytes**: - `fetch.min.bytes` — minimum data a broker waits to accumulate before responding. - `fetch.max.bytes` — soft cap on bytes returned across all partitions in one fetch response (default ~50 MB). - `max.partition.fetch.bytes` — cap on bytes returned *per partition* (default ~1 MB). The consumer pulls these byte-sized chunks and stores the decoded records in **in-memory per-partition buffers**. ### Layer 2 — the application (record count) `max.poll.records` (default **500**) controls how many records a single `poll()` returns to *your code*. The consumer drains its buffers and hands you at most that many records, regardless of how many bytes were fetched. The rest stay buffered for the next poll. ### Why decouple them? Network efficiency wants large fetches (fewer round trips). Application liveness wants small, predictable batches so each `poll`/process cycle is short. `max.poll.records` lets you fetch big but process in small bites. Example: a fetch may bring back 5,000 records; with `max.poll.records=500`, ten consecutive polls drain that buffer with **zero** extra network calls. ### Tuning guidance - **Heavy per-record processing** (DB writes, external calls): lower `max.poll.records` (e.g. 50–100) so each poll's work fits comfortably inside your processing budget and you return to poll promptly. - **Cheap processing / high throughput**: raise it. - It is the main lever for keeping poll-loop iterations bounded, which protects you from blowing the liveness window (that deadline itself, `max.poll.interval.ms`, is owned by the long-processing leaf). ### Common misconception Lowering `max.poll.records` does **not** reduce broker load or network traffic — fetches are byte-sized and unchanged. It only reshapes the in-memory hand-off. If you want fewer/smaller network fetches, tune the `*.bytes` settings instead.

  • If I set max.poll.records=10, will Kafka issue smaller fetch requests to the broker?
    No. Fetch requests are byte-sized and governed by the fetch.*.bytes settings. The consumer still pulls large chunks and buffers them; max.poll.records only limits how many buffered records each poll() hands to your application.
  • When would you lower max.poll.records?
    When per-record processing is expensive (e.g. synchronous DB writes or remote calls), so each poll returns a small batch that you can process quickly and return to poll, keeping iterations bounded and staying within the liveness window.

saying these in an interview costs you the question

  • Saying max.poll.records controls how many bytes are fetched from the broker.
  • Claiming lowering it reduces network/broker load.
  • Confusing it with max.partition.fetch.bytes (count vs bytes).
  • Thinking each poll always returns exactly max.poll.records records even when fewer are available.

context

resolving links…