How does buffer.memory and max.block.ms create back-pressure on a producer, and what does the application experience when the buffer fills?

## What buffer.memory is `buffer.memory` (default **33554432**, 32 MB) is the **total** amount of memory the producer may use to buffer records waiting to be sent — the budget for all `ProducerBatch`es across all partitions, managed by the **`BufferPool`**. It is *not* per-partition and *not* the same as `batch.size`. ## How back-pressure arises Producing is a producer/consumer system between two rates: - **Production rate** — how fast the app calls `send()`. - **Drain rate** — how fast the **Sender** thread can serialize batches onto the wire and get acks (limited by broker latency, network, `acks`, `max.in.flight.requests.per.connection`, compression, etc.). If production > drain for long enough, buffered bytes climb toward `buffer.memory`. Once the pool can't satisfy an allocation, the **append step inside `send()` blocks** on the calling thread, parked until the Sender frees a batch's memory back to the pool. ## The role of max.block.ms `max.block.ms` (default **60000** ms) bounds **how long `send()` may block** for two things combined: (1) waiting for **metadata** when the topic/partition is unknown, and (2) waiting for **buffer memory** when the pool is exhausted. If the wait exceeds `max.block.ms`, `send()` throws `org.apache.kafka.common.errors.TimeoutException` — e.g. *"Failed to allocate memory within the configured max.block.ms"*. Setting `max.block.ms=0` makes send fail-fast (never block). ## What the application experiences - **Normal:** `send()` returns near-instantly. - **Buffer filling:** `send()` calls start to **block** for milliseconds-to-seconds. Application throughput is throttled to the drain rate — this is healthy back-pressure. - **Buffer stuck full:** `send()` throws `TimeoutException` after `max.block.ms`. The record was *not* enqueued. The app must decide: retry, drop, or slow down. ## Diagnosing Producer JMX metrics: `buffer-available-bytes` (low/zero = pressure), `buffer-exhausted-rate` / `buffer-exhausted-total` (allocations that had to wait), `waiting-threads` (threads blocked in the pool), `bufferpool-wait-time-total`. Rising `record-queue-time-avg` also signals draining can't keep up. ## Remedies 1. **Raise `buffer.memory`** — absorbs longer bursts (costs heap). 2. **Increase drain throughput** — tune `linger.ms`+`batch.size`+`compression.type`, relax `acks` if durability allows, raise `max.in.flight.requests.per.connection`, scale brokers/partitions. 3. **Add producers / partitions** to parallelize. 4. **Handle it in the app** — bounded queues, async overflow handling, or surface the `TimeoutException`. 5. **Set `max.block.ms` deliberately** — small for fail-fast/latency-sensitive paths, larger to tolerate bursts. ## Key edge cases - Even with huge `buffer.memory`, a permanently slow/down broker means the buffer eventually fills — back-pressure is inevitable, only deferred. - A single oversized record can't be buffered if it exceeds available pool memory; it waits then times out. - `flush()`/`close(timeout)` also interact: they block until buffered records drain or the timeout hits.