Backpressure at the Broker

## What "saturated" means here A node is **saturated** when the work offered to it exceeds what it can complete per second. Crucially, this is not the same as a client hitting a configured ceiling. Every client can be inside its `rate ceiling`, every connection cap can be respected and every record can be under the record-size ceiling, and the node can still run out of disk throughput, handler-pool threads or network capacity — because ceilings are configured per client and saturation is the **sum** of all of them plus whatever the node is doing for itself. Backpressure at the broker is the name for whatever the node does about that gap. The reason this is asked at the first screen is that the four available responses produce four completely different symptoms at the call site, and engineers routinely read one as another. ## The four responses - **Queue the work.** The node accepts the request into a bounded internal request queue and processes it when a handler is free. This absorbs a burst well and costs only added delay — until the queue is full, at which point the design must take one of the other exits. - **Delay the answer.** The node serves the request but holds the response longer than it needed to, pacing the writer. Nothing is reported as wrong; the writer simply observes that the call took longer. - **Block the writer.** The node stops draining the connection. Transport-level flow control then pushes the pressure back into the client's own memory: unsent records pile into the **writer send buffer**, and when that is full the send call itself stops returning. - **Refuse.** The node rejects the request with an error the client has to handle. This is the only response that hands the decision back to the application in so many words. ## What each looks like at the call site | Node's response | What the writing application observes | Where the work is waiting | How records can be lost | |---|---|---|---| | Queue | Slower responses, no errors | In the node's request queue | If the queue's overflow exit discards | | Delay | Slower responses, no errors | In the node, deliberately | Not directly; the writer's buffer grows behind it | | Block | Send calls stop returning; threads pile up | In the writer's unsent buffer | If the process dies with the buffer full | | Refuse | An explicit error per request | Nowhere — it was rejected | Only if the application drops what it cannot send | Notice that the first two rows are indistinguishable from each other, and both are indistinguishable from a node that is on its way to failing entirely. That is why "writes got slower" is a starting point for an investigation and never a diagnosis. ## The quiet exit Both the node's request queue and the writer's send buffer are finite, and when either fills the design has a choice: **refuse the caller, or discard the record**. Discarding is the quietest thing in this whole subject. The send call returns successfully. No error is raised, no failure counter on the writing side moves, and a dashboard built on error rates stays green while records disappear. The loss shows up only if somebody reconciles what the application handed to the client against what the cluster actually accepted over the same interval, or reads whatever discard counter the client itself keeps. This is the answer an interviewer is listening for: of the ways to respond to saturation, the one that never fails anything is the one that loses data. ## The response you get varies by platform It is a mistake to learn one platform's habit as the model. Some designs prefer to refuse an over-capacity request outright so that the client retries; some prefer to hold the answer; some rely on transport flow control and never signal anything, so the symptom only ever appears inside the client; a rented cluster may refuse where a self-run one would have queued, because refusing is cheaper to operate at scale. When you describe this in an interview, describe the **behaviour** — queued, delayed, blocked, refused — rather than assuming everyone's node does what yours does. ## What an operator does first 1. **Name the symptom precisely.** "Writes are slower", "writes are being refused", "the sending application has stopped" and "records are missing with no errors" are four different rows of the table above. 2. **Establish which response is in play** before changing anything, because the remedies point in opposite directions: a refusing node wants less offered work or more capacity, while a stalled application wants its own buffer behaviour fixed. 3. **Reduce, then resize, then rewrite.** Take offered work away first (it is the only lever that acts immediately), add capacity second, and change what the writer sends or how it behaves third. One thing not to do reflexively is deepen the node's request queue. A deeper queue converts refusals into latency; the work still has to be done, the node still cannot do it, and now every caller waits longer before finding out.