A broker cluster is sized exactly to its measured peak rate - what routinely arrives that the machines have no room for, and what should the design target have been?

## Why a cluster at one hundred percent of its measured peak has no answer Every measurement is taken under conditions that will not always hold. A peak rate is observed with the full node count present, every reader keeping up, and writers sending each record once. Size to that number exactly and the cluster is correct on precisely the days nothing goes wrong - which is not what it is for. The **spare share** is capacity deliberately left unused at peak so that the cluster has somewhere to put the load that arrives when something does go wrong. It is a sizing decision, not an operational one: it is spent by dividing the derived demand by a target utilisation rather than by one. ## The three claimants that reliably arrive These are not the only unplanned loads a cluster sees, but they are the three that show up often enough that a sizing write-up which ignores them is incomplete: 1. **A replay.** Where records are retained and can be re-read, a consumer that resets and reads a retention span from the start generates read demand unrelated to the write rate, served largely from the volume rather than from memory holding recent records. On platforms that delete on acknowledgement there is nothing to replay - which is worth stating, because it changes the reserve those platforms need. 2. **A retry surge.** When writes slow or fail, writers re-send. The re-sent bytes cross the network and reach the accepting node whether or not anything later recognises them as duplicates, so they are real ingress. This is the claimant with a feedback loop attached: slowness causes retries, retries cause slowness. 3. **A lost node's share.** Its clients reconnect elsewhere, its traffic moves, and whichever partitions it was accepting writes for are taken over by other nodes. Nothing was added to the cluster's total work - it is simply spread over fewer machines. The reason they belong in one answer is that they **co-occur**. A node dies, so writes to it fail, so writers retry, so a consumer that fell behind during the disruption replays. The reserve has to cover the combination, not each in turn. ## The arithmetic of a lost node On a cluster of `n` evenly loaded nodes, losing one spreads its share over the remainder, so each survivor takes roughly `1/(n-1)` more than before: | nodes | extra load per survivor after one loss | utilisation that becomes | |---|---|---| | 3 | +50% | 60% becomes 90% | | 4 | +33% | 60% becomes 80% | | 6 | +20% | 60% becomes 72% | | 10 | +11% | 60% becomes 67% | Two things fall straight out of that table. **Small clusters need a much larger reserve than large ones** for the same survivability, because one node is a larger fraction of them. And if the failure unit is a whole failure domain rather than a single node, the sum is over everything in that domain at once, which for a three-domain cluster means losing a third of the machines, not one of them. ## What the design target looks like The target is stated as a utilisation at peak, derived rather than inherited: - start from the demand derived from measurement, already scaled by the peak-to-average ratio; - decide what must be survived while still at peak - one node, one failure domain, or nothing; - add the replay and retry allowance the platform's shape makes possible; - divide the demand by the resulting target fraction to get the node count; - write down what the number excludes, so the exclusion is a decision rather than a discovery. A specific percentage quoted from anywhere is worth much less than the derivation; the same target that is prudent on ten nodes is reckless on three. What does not change is the direction: the reserve exists so that the worst interval and the worst day can be the same day. ## What the spare share is not - It is **not growth headroom.** Coping with more traffic next year is a separate exercise on a separate timescale; the spare share is for load that arrives this afternoon with no notice. - It is **not a per-client allowance.** Limiting what one writer may take is a different mechanism with a different purpose; the reserve is about the machine, not about fairness between tenants. - It is **not free of cost**, and pretending otherwise loses the argument. It is bought deliberately, and the honest framing is what it buys: the difference between a degraded interval and an outage. ## The failure it prevents The shape of the incident is always the same. A cluster runs comfortably at its measured peak for months. A node is lost during the busiest interval. The survivors take its share and cross their line. Writes slow, writers retry, the retries add ingress, a consumer falls behind and later replays over the volumes, and the cluster that was exactly the right size is now the wrong size in every resource at once. Nothing in that chain is exotic; all of it was foreseeable from the sizing sheet.