A job deduplicates events by holding every id it has seen. Why does that retained set grow forever, and what bounds it?

## The retained set, and which parts of it can grow **The retained set** is everything a running job is still holding between records: running accumulators, records buffered awaiting a group, deduplication entries, both sides of a join, and pending per-key callbacks. Treat it as the job's real storage system — it has a size, a per-access read and write cost, a durability story, and, only if the author gives it one, a removal policy. Not every step retains anything. A filter or a projection looks at the record in front of it and forgets it, and can run for a decade at a constant footprint. A **deduplication check cannot**: to know whether it has seen an id before, it must still be holding that id. ## Why the logic has no natural end Written out, the check is pure insert: 1. Take the id off the record. 2. Look it up among the entries this step holds. 3. If it is present, drop the record as a duplicate. 4. If it is absent, store it and emit the record. There is no fifth step. Nothing there deletes, and no runtime in this class deletes an entry you never told it to delete. So the entry count tracks **the number of distinct ids the job has ever seen**. Over a finite input that number stops at the end of the input; over an endless one it does not. The sizing arithmetic follows, and this is the part candidates most often get wrong: - **entries** ≈ distinct ids within whatever retention you actually enforce — with no removal rule, that is all of them, since the job began; - **bytes** ≈ entries × (encoded key + encoded value + whatever per-entry overhead the store adds, which is rarely zero); - **the input rate does not appear in either line**. A million records a second over a thousand repeating ids is a thousand entries. Ten records a second of globally unique ids is ten new entries a second, forever. ## The three rules that actually remove an entry | removal rule | what triggers it | where it fits | |---|---|---| | **entry expiry** | a fixed interval has passed since the entry's last write, or in the other variant since its last read | deduplication sets, lookup-style entries, anything with a natural staleness horizon | | **per-key scheduled callback** | a moment the job registered against one key arrives, and the handler deletes that key's entry | a burst of activity from one user ended by a gap of inactivity; anything with a per-key deadline | | **cleanup on a completeness claim** | the job asserts no record older than a given moment is still expected, so entries belonging before it can go | grouped results that can be declared closed; the clock that produces the assertion is another node's subject | All three are things the author asks for. None of them is on by default in a way you can rely on. ## What differs between engines, and why it matters here This is a class of systems that disagrees about mechanism, so state the model you are describing rather than assuming one: - **Which rules are offered differs.** Some runtimes express removal as a declarative expiry attached to the entry; others give you only a callback you register and a handler you write; several offer both, and the two behave differently on restart. - **The clock differs.** An interval may be measured against the worker's own wall clock, or against the job's own assertion of how far the input has progressed. The second stops advancing when the input does, and entries then stop being removed. - **Removal may be lazy.** Some stores drop an expired entry the moment it is due; others mark it and reclaim the space only when the entry is next touched or during a background file merge, so bytes fall later than the entry count does. - **Some models have no long-lived retained set at all.** The two-phase disk-to-disk batch model writes intermediate results to disk between its two phases and keeps nothing between runs, so unbounded growth of this kind cannot happen in it. A runtime that cuts a continuous input into repeated small finite jobs touches its state once per chunk rather than once per record, which changes the cost of removal but not the need for it. ## It is not only a memory problem The first instinct is that the worker will eventually run out of room. That happens, but it is usually not the first symptom, and more memory only moves the date. Every byte in the retained set is also a byte in the periodic **durable snapshot** — the consistent copy written to storage outside the workers — and therefore a byte a restarting job must read back before it processes its first record. A job that has been fine for four months frequently announces the problem as a restart that no longer finishes. ## Saying it in an interview The answer an interviewer is listening for is two sentences of mechanism and one of sizing: the logic inserts and never deletes, so the retained set grows with distinct keys rather than with throughput; a removal rule has to be chosen deliberately, and choosing one means accepting the correctness it costs.