Producers see write latency climb while the volume shows spare space and no errors — which storage decision should you suspect?

## The symptom, and why it points at storage The reported problem is almost always at the application: producers are waiting longer to have writes accepted, and the effect grows with load. Meanwhile every storage view is reassuring — the volume has room, reports no faults and raises no alarm. That combination is characteristic rather than mysterious. A volume being made to wait is not a volume that is failing, and most storage monitoring is built to notice faults and fullness, not to notice that a healthy device is being held to a figure. The reason this lands on producers specifically is the append path. Records arrive continuously and have to be written out at the rate they arrive; when the volume cannot absorb that rate, the unwritten work accumulates and the acceptance of each write takes longer. The application feels it as latency because that is the only place it can appear. ## Provisioning shape one: throughput bought apart from size On most network-attached storage, how many bytes per second a volume may move is a separate dimension from how many bytes it may hold, and the two are often bought together in a way that hides the coupling — a default allowance derived from the requested size, or a tier chosen for its price per stored byte. The consequence for a broker: - A volume sized generously for retention can still have a modest throughput allowance. - The allowance is usually a hard figure, so behaviour is *flat*: fine below it, and degrading immediately above it. - Growing the volume to go faster sometimes works and sometimes does nothing, depending on whether the allowance is tied to size at all. ## Provisioning shape two: the volume is shared The other shape is contention. The broker's volume may be shared with a batch job, with logs and telemetry written by the same host, with a second record-serving node placed on the same underlying device, or with another tenant entirely. Here the symptom is *not* flat: the same broker load is fast at one hour and slow at another, because the available throughput moves with a schedule you do not control. ## Why it does not look like a storage problem | what the storage view reports | what the application experiences | |---|---| | space used well under the volume's size | producer write latency climbing with load | | no device faults, no degraded state | the slow tail growing faster than the average | | service times "within specification" for the tier | timeouts and retries at the client, under load | | throughput at its allowance, reported as normal | acceptance of each write taking noticeably longer | ## Telling the two apart 1. **Plot the broker's own write rate against the observed write service time over several days.** A hard allowance shows a knee at the same figure every time. Contention shows no consistent knee. 2. **Check whether the slow periods correlate with the broker's load or with the clock.** Correlation with the clock, or with another team's schedule, points at a neighbour. 3. **Compare record-serving nodes.** If one node is slow and its peers carrying similar load are not, look at what else that node's volume is shared with rather than at the broker's settings. ## What varies between platforms - A platform that **retains** records writes the full incoming rate to the volume continuously, so a throughput shortfall is visible almost immediately under load. - A broker that **removes records on acknowledgement** and is keeping up may barely touch the volume at steady state, so the same shortfall stays hidden until readers fall behind or the process restarts and has to read back. - A design whose primary store is a **remote service** replaces the per-volume allowance with a service's own rate and error behaviour — the shape of the problem is the same, the place you look is not. ## What this is not It is not a volume that has filled up, which has its own symptom and its own remedy; the stem rules that out explicitly. It is not a durability setting, which decides what the write waits *for* rather than how fast the media can absorb it. It is not a question of how large the volume should be, which is arithmetic from the incoming rate. And it is not, at this stage, an alerting question — deciding what to alert on when service times rise is a separate discipline. What it is, is a storage *character* decision made before traffic arrived and never revisited: the volume was chosen by price and size, and nobody wrote down what sustained rate the append path was going to need from it.