The age bound was cut from seven days to one to free space, yet the volume is still full ten minutes later — why?
answer
- policy change, not a delete
- a background pass on its own schedule
- whole segments, never the active one
- eligibility beats age
- unlinked and closed before space returns
basics
~20 sChanging an age bound marks records removable; it does not remove them. A background removal pass does that on its own schedule, takes whole closed segments, never touches the active one, and cannot touch bytes something still pins.
solid answer
~50 sAn age bound is a policy, not an operation. The work is done by a background **removal pass** that runs on its own schedule, so the change has an effect when the pass next runs rather than when you save it. The pass also works at **segment** granularity: a closed segment goes only when none of its records is still inside the bound, so a segment mixing old and recent records survives entirely, and the **active segment** — the one still being appended to — is never removed at all. On top of that, some bytes were never eligible: records a hold protects, and, on platforms that bound a per-subscriber backlog rather than the stream, bytes a subscriber that has not advanced still holds. Finally, space returns to the filesystem only once the file is actually unlinked and nothing holds it open.
go deeper
Recall that a retention setting decides what may be removed, not what is removed now. Something in the background does the removing, and it runs when it runs.
Explain segment granularity: whole closed files, never the active one, and only when no record inside is still within the bound. That alone explains most of the delay and all of the staircase.
Separate 'not yet' from 'never'. Under incident pressure the question is which bytes are ineligible and why, because no bound you set will move them, and you have already paid for the cut.
Note the design coupling: segment size trades write efficiency against how responsive a retention change can ever be, and that trade is made long before the incident that tests it.
## A bound is a policy, not an operation An **age bound** is the maximum age a record may reach before it becomes *removable*. Editing it changes eligibility. It does not walk the volume and delete anything. The deletion is done by a **removal pass**: background work the store schedules for itself, deliberately unhurried, because removing history is irreversible and costs I/O on a node that is usually busy. So the first answer to "why has nothing happened" is often simply that the pass has not run since the change. That also means the relief is **stepwise, not smooth**. Free space does not creep upward; it jumps when the pass runs and then sits flat again. ## The pass removes whole segments A store rolls its append files closed into **segments**. The one still being appended to is the **active segment**. - A closed segment becomes removable only when **none** of its records is still within the bound — in practice judged from the newest record it contains. A segment holding both six-day-old and one-hour-old records survives a one-day bound in full, including its six-day-old records. - The **active segment is never removed**. It has to roll closed first, and it rolls on a size or age trigger. On a busy stream that is minutes; on a quiet one it can be hours, and on a nearly idle stream the entire stream may be a single unremovable file. - Therefore **the history still readable is always more than the bound**, by up to a segment per scope. That is not a bug; it is the granularity the design chose so that removal is a cheap file operation rather than a rewrite. A large segment size is a good trade for write throughput and a bad one for the responsiveness of a retention cut, and an operator who has cut the bound during an incident discovers that relationship at the worst moment. ## Bytes that were never eligible at all | What you see | What it usually is | |---|---| | Free space unchanged for minutes | The removal pass has not run since the change | | Space frees in large steps | Removal takes whole segments, not records | | History older than the bound still readable | Its segment still holds a record inside the bound, or it is the active segment | | One stream never frees anything | Its bytes are pinned by something, not merely old | | The store says it removed, the filesystem disagrees | The file is unlinked but still held open, or unlinking is deferred | The "pinned" row is the one that turns a puzzle into a diagnosis. Two common sources: 1. **A hold that suspends removal.** A rule can declare that some records must not be removed regardless of the configured bounds. Shortening the bound has no effect on them by design. 2. **A subscriber that has not advanced.** Platforms differ sharply here, and the difference decides what you are looking at. Where the bound applies to a **per-subscriber backlog**, unread bytes stay ineligible for as long as that subscriber exists, so an abandoned subscriber is an unbounded byte consumer and a retention cut will not touch its backlog. Where the bound applies to **the stream itself**, the opposite is true: the pass removes on schedule whether or not anyone read the records, and the reader simply loses them. Know which shape you are operating. ## Space has to come back from the filesystem too Even a genuinely removed segment does not necessarily return bytes immediately. A file's space is reclaimed when it is unlinked *and* no process still holds it open; some stores also rename a segment first and unlink it after a grace interval, precisely so that a mistake is recoverable for a short time. Until then the store's own accounting and the operating system's free-space figure disagree, and only one of them decides whether the next write is accepted. ## What the cut already cost **The retention window** — the span of history still readable right now — *is* the replay budget. Shortening it is the one action in this whole incident that destroys something permanently: every record the pass then removes is gone from this store, and the reduction applies from now on, so the investigation that follows the incident and any rebuild that needs to re-read history both get a smaller budget than they had an hour ago. Paying that price and then discovering the volume is still full is the worst of both outcomes, which is exactly why the shape of the removal pass is worth knowing before the incident rather than during it.
- Why is history older than the configured age bound still readable even after the pass has run?Because removal is per segment. A closed segment goes only when none of its records is still within the bound, so one recent record keeps the whole file — and its old records — alive. The active segment is never removed at all. The surviving history is always a bit more than the bound.
- Does a shorter age bound apply only to records written after the change?No. The bound is evaluated against the records that exist, so older records become eligible immediately. What is delayed is the pass that acts on that eligibility, and what blocks it entirely is anything pinning those bytes.
- Why is free space rising in large jumps instead of falling steadily?Because the unit of removal is a whole segment. Each pass either takes a file or does not, so the free-space graph is a staircase. A smooth curve would imply record-by-record deletion, which no store built for sequential append does.
saying these in an interview costs you the question
- Thinks saving a shorter bound deletes records synchronously
- Expects removal record by record, so free space should fall smoothly
- Assumes a new bound applies only to records written after it
- Believes the active segment is removed like any other
- Says pinned bytes will go once they are old enough
- Trusts the store's accounting over the filesystem's free space