The data volume under a broker node is full — what happens to writes landing on that node, and which streams feel it?

## What a full data volume means to an append-only store A broker or streaming platform holds records by **appending** them to files on a **data volume** — the block device the node writes to. The store rolls those files closed at some size or age; each closed unit is a **segment**, and the one still being appended to is the **active segment**. Every accepted write needs room on that volume, and so does the bookkeeping the store keeps beside the records. When free space reaches zero — or reaches whatever floor the platform reserves for itself so it can still function — there is no partial answer available. The store cannot write half a record and remain an append-only store, and it cannot make room on the write path, because removing history is governed by a policy and a background pass, not by pressure. ## Refusal, not degradation This is the distinction interviewers listen for. A full volume is not congestion that clears when a burst passes. Nothing a writer does will make room. The condition persists until an operator changes something. Designs differ in how the refusal reaches the client, and saying so is part of a good answer: - some platforms flip the node, or every stream held on the exhausted volume, into a state where writes are rejected with an explicit error while reads continue; - some fail each individual write attempt and leave the writing application's retry policy to decide what happens next; - some hold the caller until its send timeout expires, so the application sees a hang rather than an error and only discovers the cause from the node's own signals. In all three shapes the answer to "can this write be accepted?" is no, and retrying harder makes it worse: retries multiply request load on a node that can accept none of it. ## The blast radius is the volume, not the stream | Question an operator asks | What is true | |---|---| | Which streams stop accepting writes? | Every stream whose data lives on the exhausted volume, whichever one grew | | Do reads stop? | Usually not — serving bytes already on the volume needs no new space | | Is anything already stored lost? | No; exhaustion refuses new writes, it does not discard old records | | Does adding writers or retries help? | No — it adds load to a node that can accept nothing | | Whose incident is it? | Everyone's on that node, which is why it escalates faster than teams expect | This is the operational point of the whole subject. One ungoverned stream, one backfill nobody warned about, one team's experiment — and the neighbours that were sized correctly go down with it. The mitigation is therefore a capacity story about the volume, not a tuning story about the stream that misbehaved. ## The read path is not entirely safe "Reads keep working" needs one qualification. On platforms where a reader's progress is recorded back onto the cluster — sometimes into a stream of its own that lives on the same volume — a node refusing writes also refuses to record that progress. Readers then appear healthy, consume records, and re-consume the same records after a restart, because nothing durable recorded how far they got. On platforms where progress is held client-side or in an external store, that symptom does not appear. Expect the difference rather than assuming either. ## Why the volume filled Exhaustion is almost never a surprise in the arithmetic; it is a surprise in **eligibility**. The bytes on the volume are the bytes the store was not allowed to remove: history still inside its configured bounds, plus anything a rule or an un-advanced reader pinned there, plus whatever else shares the device. A volume that several streams share with no bound on any of them has no defence at all — the sum of what may be stored is simply unbounded, and the first stream to grow decides everyone's fate. ## What it costs to fix in a hurry The fastest way to restore writes is to shorten **the retention window** — the span of history still readable on the store right now. That window *is* the replay budget: everything a reader could still be rewound into, everything an investigation could still read, everything a rebuild could still consume. Shortening it converts history into free space immediately and permanently. That makes it powerful and makes it the last rung, not the first: relief that does not destroy history — more space, moving the pressure elsewhere, releasing bytes something is pinning — is tried before the one action that cannot be taken back. A junior answer that gets "it refuses writes, for everything on that volume, and reads generally survive" is already a good answer. The rest is what seniority adds.