A cluster sized from a daily average of 8,000 records per second sees 40,000 in its busiest ten minutes - which number should have sized it?

## What the ratio is The **peak-to-average ratio** is the multiplier between the rate a cluster sustains over a representative interval and the rate it carries in its busiest one. In the example it is: ``` sustained rate 8,000 records/s busiest ten minutes 40,000 records/s peak-to-average 40,000 / 8,000 = 5x ``` It is the single number that separates a cluster that runs from a cluster that runs on average. Most real traffic has one: business hours against nights, a batch job that opens at the top of the hour, a mobile client population that wakes on a push, a retry surge after a dependency recovers. ## Why the average cannot size the thing that fails Saturation is not an average phenomenon. An interface that can move 1 GB/s does not move 5 GB for one second and catch up later; the excess queues, and when the queue is full the excess is refused, dropped or delayed. A cluster sized at the mean is, by construction, over its line for as long as the busiest interval lasts. What that looks like from outside **differs by platform**, and saying so is part of the answer: - on some platforms the writes are accepted but acknowledged more slowly, so the visible symptom is climbing write latency and writers holding records in memory; - on others the request is refused outright with an error and the writer must retry, which adds its own traffic to the interval that is already the worst one; - on a rented cluster the ceiling may be enforced as a policy rather than reached as a physical limit, and the surplus is refused or slowed by rule. In every case the second-order effect is the same shape: **writers that cannot hand records over start buffering, and a buffer that fills either blocks the application or discards.** The incident is rarely reported as "the cluster was busy"; it is reported as the application being slow. ## Which numbers take the peak and which take the average This is the distinction most sizing exercises get wrong in one direction or the other - either everything is sized for the peak, which buys volumes nobody needs, or everything is sized for the mean, which buys a cluster that fails daily. | quantity being sized | which measurement | why | |---|---|---| | network interfaces, in and out | busiest interval | saturation is instantaneous; there is no catching up | | request handling and per-record work | busiest interval | the same records arrive in the same second | | traffic between record-serving nodes | busiest interval | it tracks ingress, so it peaks with it | | bytes stored for a retention span | sustained rate | storage integrates the rate over the whole span | | the reserve left free on the machines | applied to the peak number | the reserve exists for the worst interval, not the mean | A worked contrast: at 8,000 records per second of 2 KB, a seven-day retention span with three copies stores roughly `8,000 x 2 KB x 604,800 s x 3`, and ten minutes at five times the rate adds well under one percent to that. The same ten minutes decides the entire network and node count. ## The averaging interval is itself a decision A rate is meaningless without the interval it was averaged over, and the interval has to match what gives way: 1. **Sub-second bursts** are usually absorbed by buffers and rarely size anything on their own. 2. **Seconds to a few minutes** is where interfaces, request handling and the write path actually saturate - this is the interval to take the maximum over. 3. **Hours and days** are the wrong granularity entirely for machine sizing, though they are the right one for stored bytes. The classic misread is a dashboard averaging over a minute that shows comfortable room while writers time out every few seconds inside it. Nothing is wrong with the measurement; it is answering a different question from the one being asked. ## Getting an honest number - Take the **maximum** of the short-interval rates across a representative period, not the mean of them, and not the single highest spike of all time unless that spike must be survived. - Measure the **write rate and the read rate separately**, because their busiest intervals need not coincide - a nightly re-read can peak when writes are at their quietest, which is good news you can only see if you looked. - Record the **record rate and the byte rate** at peak, since a burst of small records and a burst of large ones stress different resources. - Expect the ratio to **change with the traffic mix**, and re-measure after a launch rather than inheriting last year's multiplier. - Treat a peak that is mostly retries as a **different problem** from a peak that is mostly new records; the first can shrink when the underlying cause is fixed.