Your team is adding a removal rule to a long-running job's retained set. What correctness does each rule cost, and what decay is it buying off?

## The shape of the trade A removal rule is not housekeeping. Every entry you remove is a fact the job used to know and now does not, and the next record that would have used it behaves as though it had never been seen. So the question is never "should we bound the retained set" — it is **which wrongness do we prefer, and can we state its size**. Two properties make this trade unusually easy to get wrong: - the cost of removal lands on **rare records** — the late retry, the straggling join partner, the user who came back after a long gap — so it is invisible in aggregate dashboards and in every test; - the cost of not removing lands on **every record and every restart**, but slowly, so it is invisible until the month it is not. ## The price list | what the entry was for | what removing it too early costs | how the failure shows up | |---|---|---| | a deduplication key | a redelivery after the interval is emitted a second time | a double charge, a double shipment, a count that is slightly high | | one side of a buffered join | the partner arrives and finds nothing, so the match is never produced | silent: a missing row, no error and no log line, usually a small percentage | | a burst of activity from one user, ended by a gap | the burst is cut in two and counted twice | session counts inflate, average duration falls, and both look plausible | | a running accumulator for a key | the total restarts from zero on the next record for that key | a metric that dips to a small value and climbs again | | a pending per-key wake-up | the thing it was going to emit is never emitted | an absence: the closing summary or the timeout alert simply does not appear | The join row is the one to name in an interview, because it is the only failure in the table with **no signal at all**. A duplicate is visible downstream, a halved burst shows in the numbers, a reset accumulator is a visible dip. A match that was never produced looks exactly like a match that never existed. ## Cleanup driven by a completeness claim The third removal rule drops entries once the job asserts that no record older than a given moment is still expected. That assertion is a **claim, not a fact** — the clock that produces it is another node's subject, but its consequence here is direct: entries are dropped on the strength of an estimate, and anything that arrives afterwards is treated as new. The same record that would have been recognised as a duplicate an hour ago is now a first sighting. Choosing this rule means accepting the estimate's error rate as your correctness budget. ## The decay on the other side What the rule buys is the absence of a slow, compounding degradation: - the **durable snapshot** — the periodic consistent copy written to storage outside the workers — carries more bytes each cycle, so taking it takes longer and interferes more with processing; - a restarting job must read the retained set back before it handles its first record, so the restart that took two minutes in week one takes an hour in week sixteen; - per-access cost rises as the working set outgrows whatever memory the store had for it, in a way that depends on the store model: entries held as ordinary live objects press on the worker process's managed memory; an **embedded on-disk store with a memory cache** starts missing its cache and pays an encode, a decode and a local file read per access; **state rewritten as files each cycle** writes a larger file set every cycle. The endpoint is the same in every model: a job that can no longer be restarted inside the window operations has for it, at which point the only fast remedy is discarding the retained set — which is a **100% loss of exactly the facts the removal rule would have shed a few per cent of**. ## How to choose the interval Derive it from the tail of the real world and then say what it costs out loud: 1. Find the horizon that matters — the longest redelivery the source can produce, the longest realistic delay between two sides of a match, the inactivity gap that genuinely ends a burst. 2. If the set at that horizon is affordable, set the interval there and **write the weakened guarantee into the contract** the consumers depend on. 3. If it is not, do not quietly shorten the interval to fit the memory: argue the horizon down with the people who depend on it, or move the remembered facts to storage built for that volume, deliberately and with its own costs. ## What varies Which rules are available, whether removal is eager or lazy, and whether a rule can emit before it deletes all differ across this class of engines. A model that keeps nothing between runs — the two-phase disk-to-disk batch model — has none of these prices and none of this decay, which is why lifting a batch job's logic into a continuous one introduces a question the original never had to answer.