Across many writing applications sharing one cluster, where should overload surface, and what do you require of every writer?

## Two places the pressure can land When an estate's writers collectively offer more than a shared cluster can absorb, someone absorbs the difference. There are two candidates, and the choice between them is a policy decision rather than a technical one. | Pressure lands in | What it looks like | Who feels it | What goes wrong | |---|---|---|---| | The cluster | Deeper request queues, later answers | Every tenant, including the innocent | The shortfall is invisible; latency has no owner | | The writers | Bounded waits, explicit failures | The team that produced the load | Noisy, and every writer needs a plan | The instinct is to protect the applications by absorbing everything in the cluster, and it is usually wrong at estate scale. A deeper queue does not create capacity; it just makes everyone wait longer before finding out, and it spreads one writer's burst across every other tenant on the same nodes. Worse, it removes the only signal that would have caused the offending team to change anything. The opposite policy — pressure surfaces in the writer that caused it — keeps the cost attached to its owner. ## What you require of every writer The rule set is deliberately short, because a rule nobody can check is decoration: 1. **A bounded unsent buffer.** Unbounded client-side buffering is a rate mismatch turning into a process death at an unpredictable moment, with every record in it lost. 2. **A cap on how long a send may wait.** An uncapped wait makes the application's availability a function of the cluster's. With a cap, a stalled cluster becomes an error the code decides about. 3. **A declared loss policy, per stream and not per service.** Most services write both measurements and business facts. One writer for both means the weakest record's policy governs the strongest. 4. **A defined destination for records that cannot be sent.** Fail the user's operation, hold the record durably on the writer's own side, or knowingly discard and count it — but pick one in advance. 5. **Published counts of what was discarded or failed.** It is the only cheap evidence that the policy is being exercised, and it costs the writer almost nothing. ## The second-order effects to predict This is where the judgment is actually tested, because each choice pushes the problem somewhere else. - **A writer that blocks becomes an outage.** If its publishing path shares threads with its user-facing path, cluster slowness becomes unavailability of the product. That coupling, not the cluster, is what takes the business down. - **A writer that fails needs somewhere to put the record**, or it has merely relocated the discard and made it look principled. - **A writer that is throttled or slowed becomes someone else's growing pile of unsent work.** The pressure does not evaporate when you push it upstream; it sits in a queue at the edge of your system, and somebody needs to know how long it can sit there. - **A cluster that answers slowly is indistinguishable from one that is failing.** Every writer reacting to slowness will treat it as a failure eventually, usually by retrying, which adds load to the thing that was already saturated. ## Agree the shedding order before you need it The single most valuable artefact is a short, boring list: which writers are asked to stop first when the cluster cannot take everything. It needs three things to be usable in an incident — a named owner per stream, a rough statement of what stops working if that stream pauses, and a pre-agreed order. Without it, the decision is taken at three in the morning by whoever has access, and the criterion becomes whichever writer is easiest to turn off rather than whichever matters least. ## Where the policy stops Two boundaries are worth stating out loud, because a lead is expected to know when this lever has run out. - **Backpressure policy cannot fix a cluster that is simply too small.** Every rule above governs the behaviour of the shortfall, not its size. If the shortfall is permanent, the answer is capacity or less work, and continuing to tune writer behaviour is displacement activity. - **It cannot fix a writer that should not be on this cluster.** When one workload's shape repeatedly saturates the shared nodes, the remaining lever is separation rather than policy, and that is a different conversation about what the estate looks like. And one caveat about portability: platforms differ in what they even offer here. Some will refuse over-capacity requests, some hold the answer, some rely on transport flow control and surface nothing at all, and a rented cluster may expose none of its internal queueing to you. Write the estate rules in terms of **what writers must do**, which you control everywhere, rather than in terms of what the node will do, which you do not.