What is max.partition.fetch.bytes, how does it relate to fetch.max.bytes, and what failure mode arises if a record exceeds it?

## Per-partition and total fetch byte caps A fetch response can pull from many partitions at once. Two settings bound its size. ### max.partition.fetch.bytes (default 1 MB) The maximum bytes returned **per partition** in a single fetch. With N assigned partitions, the naive worst case is roughly N × max.partition.fetch.bytes — which is why a second, overall cap exists. ### fetch.max.bytes (default ~50 MB) The maximum bytes returned **across all partitions** in one fetch response. Introduced by **KIP-74** precisely so that consumers subscribed to many partitions don't blow up memory: it bounds the whole response, while max.partition.fetch.bytes still keeps any single partition from monopolizing the response. ### Both are SOFT limits — the progress guarantee Kafka stores records in compressed **record batches**. A limit could be smaller than the first batch on a partition, which would mean the broker returns zero usable data → the consumer can never advance → permanent stall. To prevent this, the rule is: **the broker always returns at least one full record batch for the first non-empty partition**, even if that batch exceeds the byte cap. Hence the caps are soft. This guarantees liveness regardless of message size. ### Failure mode, then and now - **Historically / conceptually**, people feared: "if a record is bigger than max.partition.fetch.bytes the consumer gets stuck." In strict early implementations this was a real risk. - **Today**, thanks to the oversized-batch guarantee, a single large message is still delivered — the consumer makes progress, just with a response larger than the soft cap for that fetch. - The remaining real-world failure is **misconfiguration**: setting max.partition.fetch.bytes well *below* the topic/broker `message.max.bytes` (a.k.a. `max.message.bytes`) wastes the soft-cap intent and can cause large per-fetch responses or memory pressure. Best practice: max.partition.fetch.bytes ≥ broker max message size. ### Why both settings exist (interplay) - `max.partition.fetch.bytes` ensures **fairness** — one hot partition can't starve others by filling the whole response. - `fetch.max.bytes` ensures **bounded total memory** — protects the client heap across all partitions. - Together they shape per-fetch memory: total response ≤ ~fetch.max.bytes, each partition ≤ ~max.partition.fetch.bytes (both modulo the oversized-batch exception). ### Tuning notes - Many partitions + small max.partition.fetch.bytes → more even round-robin draining across partitions per fetch. - Large records → raise max.partition.fetch.bytes to keep responses efficient and avoid relying on the single-batch override every time. - Watch client heap: prefetch buffers hold up to these byte sizes per in-flight fetch per broker.