A job running four months now takes an hour to restart instead of two minutes, and per-record latency has doubled. How does an unbounded retained set cause both?

## Why the two symptoms are one cause **The retained set** is everything the job is still holding between records: accumulators, buffered records, deduplication entries, join sides and pending per-key wake-ups. If nothing removes entries, that set grows with distinct keys for the life of the job. Every one of its bytes is charged twice — once to the periodic **durable snapshot**, the consistent copy written to storage outside the workers, and once to the store the workers read on every access. Restart time is the first charge; per-record latency is the second. The defining property is that neither charge announces itself. There is no failure, no error rate, no threshold crossed — just a job a little slower every week and a restart a little longer, until an ordinary deployment does not finish inside its window. ## Why coming back takes longer - A restarting worker must have its entries in place **before** it processes a record, so the read is on the critical path. - The bytes it reads back are the bytes the retained set holds, which is why restart time follows the same curve as entry count. How much has to be read varies with the store model: where a snapshot is written whole each cycle, it is the full set; where snapshots are incremental, a restore reads a base plus the increments since, which is smaller per cycle but still grows with the base. - More of it is fetched from shared storage across the network at once, by every worker simultaneously — a different bottleneck from the steady state, and why restart time can degrade faster than the set grows. - Where a job is stopped deliberately, the **planned snapshot** taken on the way down has the same size problem: the graceful stop that used to take a minute now takes long enough that people start killing it instead, which is how a clean shutdown turns into a recovery. ## Why each record costs more, by store model | store model | what grows | what the record pays | |---|---|---| | **in-process memory store** — entries kept as ordinary live objects inside the worker | the live object graph in the worker process's managed memory | more time lost to the process's own memory management, and eventually a hard ceiling at that process's limit | | **embedded on-disk store with a memory cache** — recent entries in memory, the rest in local files merged in the background | the share of accesses that miss the cache | an encode, a decode and a local file read per miss, plus a background file merge competing for the same disk | | **state rewritten as files each cycle** — each cycle writes changed entries to a versioned file set, with a periodic full rewrite | the size of each cycle's file set and of the periodic full copy | a larger fixed cost per cycle, paid whether or not many records arrived in it | The first model degrades suddenly and the second gradually; the third degrades in steps at each full rewrite. Naming which model you mean is the difference between an answer that is true and one true only of the engine you used last. ## Why it only appeared in month four - The growth is linear in distinct keys, but the **cost curves are not**: an access is flat until the working set exceeds the cache, then steps up; a restart is fine until it exceeds the operational window, then it is an outage. - Tests never see it. A test runs for seconds over a handful of keys, and the defect is defined by months and millions of keys. - The early symptoms are attributed elsewhere — a slow week is blamed on traffic, a slow restart on the storage layer — because nothing in the job changed. ## Confirming it is the retained set and not something else The distinguishing evidence is that the trend is monotonic across restarts and independent of input rate: latency does not fall when traffic falls, and each restart is slower than the one before it even when the intervening period was quiet. A slowdown that tracks traffic is a capacity story; a slowdown that tracks *calendar time since the job started* is this one. Comparing the size of successive snapshots over weeks settles it, and which numbers a team watches routinely is another node's subject. ## Getting out of it 1. Add the removal rule. It stops the curve but does not by itself rewind it. 2. Expect the bytes back later than the entries. Some stores remove eagerly; others mark the entry and reclaim space on its next touch or during a background file merge, so a set of entries never touched again can sit on disk long after they logically expired. 3. If space does not come back in time, the remaining options are to rewrite the stored snapshot outside the running job — reading it as an ordinary dataset, dropping what should have expired and writing a new one the next run reads — or to discard the retained set and rebuild from the source, which costs exactly the facts you were keeping. 4. The two-phase disk-to-disk batch model never has this problem, since it keeps nothing between runs: a continuous job inherits a maintenance obligation its batch predecessor did not have.