An idle sweep cleared a stream after a fourteen-day observation window and the quarterly close broke, so what was wrong with that window?
answer
- silence is not evidence yet
- longest gap a legitimate user leaves
- sweep schedule is not the window
- waiting is cheap, acting early is not
- declared cadence buys a shorter window
basics
~20 sThe observation window was shorter than the interval between legitimate uses. A quarterly reader is absent for about eighty-nine days out of ninety, so fourteen days of silence is the normal state and proves nothing. The window must contain the slowest periodic user, with margin.
solid answer
~50 sAn observation window converts silence into evidence, and it can only do that if it is longer than the longest gap a legitimate user leaves. A quarterly job is absent for almost the whole quarter by design, so a fourteen-day window records its normal state and reads it as death. Size the window from the slowest known periodic user — a quarter plus margin is a common floor — rather than from how often the sweep happens to run. The cost of a longer window is narrow: the candidate goes on occupying room and being paid for while it waits. The cost of a short one is a broken periodic job and a step that cannot be taken back. You shorten the window honestly by having owners declare a use cadence at creation, not by hoping.
go deeper
Remember that some jobs run monthly or quarterly, so a stream can be legitimately silent for a long time. Judging it over two weeks measures the normal state of a slow job rather than its death.
Explain that the window must exceed the longest gap a legitimate user leaves, and that the sweep's schedule is a separate number. Add margin for jobs that slip and for gaps in the observation itself.
Argue the asymmetry explicitly: waiting costs room and the bill for stored bytes, acting early costs a broken close and a step with no way back. Then show how declared cadences let you shorten the window honestly.
Set the default window for an estate that runs jobs you have not heard of, and design the incentive that makes teams declare their cadence rather than leaving everything on the slow default.
## Why silence needs a window An idle sweep does not observe disuse. It observes **silence**, and a window is what converts silence into evidence. The conversion is only valid when the window is longer than the longest gap a legitimate user leaves between uses. Below that threshold the sweep is measuring its own impatience. A quarterly regulatory extract is the classic counter-example. It is absent for nearly the whole quarter by design. Any window shorter than a quarter records that absence faithfully, and the finding is indistinguishable from a stream nothing will ever touch again. ## Sizing the window The window is a property of the **estate's slowest legitimate user**, not of the sweep's schedule. Run the sweep as often as you like; the window it reports over is a separate number. - Start from the slowest periodic pattern the organisation actually runs: month-end, quarter-end, half-year, annual. - Add margin for the job that slips — a quarter-end extract that runs a fortnight late is still legitimate. - Add margin for the sweep's own gaps: a window is only as good as the continuity of the observation behind it, and a monitoring outage in the middle of it is a hole. - Treat anything the sweep has never seen used at all as the same case as a long-idle stream, not a stronger one; a stream created in error and a stream used yearly look identical for the first eleven months. ## The asymmetry that sets the number | | Window too long | Window too short | |---|---|---| | What happens | Candidates sit longer before anything is done | A live periodic user is declared dead | | What it costs | Room on a cluster and the bill for the bytes it keeps | A broken close, a failed extract, and a step with no way back | | How it is discovered | On the next sweep, at no risk | By the user, at the worst possible moment | | How it is repaired | Wait less next time | Often not repairable at all | The two columns are not symmetric, and that asymmetry is the whole argument. Waiting is cheap and reversible; acting early is neither. The cost of waiting is real, though — an orphan goes on paying for the bytes it stores and on occupying room on a shared cluster — which is exactly why the window should be *sized*, not simply made enormous. ## Making a long window affordable The way out of the trade is to stop treating every stream as equally mysterious: 1. **Declare a use cadence at creation.** If the owner states that a stream is written and read continuously, a short window is legitimate for it. If they state it is read quarterly, the sweep knows what silence means and can wait. 2. **Run two tiers.** Streams with a declared continuous cadence are judged on a short window; everything undeclared falls back to the long default. This gives teams a reason to declare, because the alternative is their streams sitting on a candidate list for months. 3. **Act on the owner record first.** Long before the window closes, an owner record that no longer resolves is actionable in its own right — that is a governance finding, and it does not need the full window behind it. 4. **Separate the finding from the action.** The sweep publishes evidence and opens a challenge; nothing irreversible is attached to the finding itself. Then a generous window costs only patience. ## What a short window is really buying Almost always, a short window is chosen because someone wants the estate cleaned up this quarter. That is a real pressure, and the answer is not to argue against it but to redirect it. A short window does not make the cleanup faster; it makes it *riskier*, and one broken quarterly close will cost more attention than every byte the sweep would have reclaimed. What genuinely accelerates cleanup is the population, not the window: streams whose owner record does not resolve, streams created in error that have never seen a single write since birth, and streams a team has explicitly disowned. Those need no waiting at all, because the evidence gathering them is not silence. ## Where this stops This is the window and what makes silence mean something. How long records survive inside the stream while it is alive is a different setting with a different purpose, and everything that happens after a candidate is accepted — the move to a replacement and the removal itself — belongs to the retirement of a stream rather than to the sweep.
- How can the estate shorten the window without risking the periodic reader?Have the owner declare a use cadence when the stream is created, and run two tiers off it: declared-continuous streams are judged over a short window, everything undeclared over the long default. Teams then have a concrete reason to declare, and the sweep no longer has to assume the worst about every stream.
- Which findings do not need the full window behind them?Ones where the evidence is not silence. An owner record that no longer resolves, a stream that has never taken a single write since it was created, and a stream a team has explicitly disowned are all actionable on their own terms. Waiting adds nothing to any of them.
saying these in an interview costs you the question
- The window should match how often the sweep runs
- Two weeks of silence is plenty for any stream
- A quarterly job would show up in some monitoring anyway
- A longer window costs nothing at all
- A stream never used since creation needs the same wait
- If the sweep missed a week, the window is still intact