Every thread in a producing application is stuck inside a send call while the broker reports no errors — why?

## Where the records actually are A send call in a messaging client is almost never a synchronous trip to the cluster. The record is appended to a **writer send buffer** — a bounded region of the client process's own memory — and the call returns, usually handing back something the caller can wait on later. A background path pulls records out of that buffer, forms them into batches, and ships them. So the send call's latency reflects how fast records *enter* the buffer, and the cluster's health reflects how fast they *leave* it. This arrangement is stable only while the drain rate is at least the fill rate. The moment the node is saturated, the drain rate falls, the buffer's occupancy climbs, and the application keeps filling it at exactly the rate it always did, because nothing has told it otherwise. ## What a client does when the buffer is full When there is no room for the next record, the client must do one of three things: 1. **Block the calling thread** until room appears. The application stops, and it stops silently — no error, no exception, nothing in the failure counters. 2. **Fail the call** immediately, so the application is told that the record could not even be accepted locally. 3. **Discard the record** and report success, which trades the stall for invisible data loss. Most clients make this configurable and many allow the block to be bounded, so the call waits up to some maximum and then fails. Designs differ on the default, and it is worth knowing which one your writers are running before an incident rather than during it. ## Why this reads as an application outage The failure mode that makes this a standard interview question is the coupling between the two thread pools. In a typical service, the threads that handle inbound user requests are the same threads that publish records — an order is accepted, and a record about it is sent. If the send call blocks, those threads are parked inside it. The service stops accepting new work, its own inbound queue backs up, its health endpoint may start failing, and an orchestrator may well restart it — which throws away everything already sitting in its buffer. | What is true | What it looks like from outside | |---|---| | The cluster is up and serving | No cluster alert fires on failed writes | | No send has failed | The application's error rate is flat | | The application's threads are parked | Requests time out; the service looks dead | | Unsent records are in client memory | A restart loses exactly those records | So "the broker slowed down" and "our service went down" can be the same event, and the causal arrow points from the cluster into the application through a buffer that nobody thought of as a queue. ## Telling it apart from the alternatives Three situations produce three different pictures, and distinguishing them takes seconds once you know what to look at: - **The node is refusing requests.** Send calls return quickly, and errors appear per request. The application is healthy and complaining loudly. - **The node is queueing or delaying.** Send calls still return quickly, because they only reach the buffer. What grows is the time between a send returning and the record actually being acknowledged, and the buffer's occupancy. - **The writer's buffer is full.** Send calls stop returning at all. Thread dumps show threads parked in the send path, and buffer occupancy sits at its ceiling. The third is the only one where the writing application is itself the casualty. ## What to do about it In the moment, the lever that acts fastest is reducing offered work — shed or pause the lowest-value producers so the buffer can drain. Capacity added to the cluster helps, but not instantly. Afterwards, the fixes are structural: - **Bound the wait.** A send that can block forever is an unbounded dependency on the cluster's health. Cap it, so the failure becomes an error the application can reason about. - **Decouple the pools.** Do not publish from the threads that serve users, or accept that cluster health is your availability. - **Decide what happens to records you cannot send** before it happens: fail the user's operation, hold them somewhere durable of your own, or knowingly drop them. - **Resist simply enlarging the buffer.** A bigger buffer buys time proportional to the size of the gap, not a fix for the gap; it also means more records lost if the process dies, and a longer delay before anyone notices anything is wrong.