Long-Lived State

Key-bound state is addressable only under the grouping key of the record in hand. Each key's entry lives with the worker that owns that key, and the step is handed no way to reach another key's entry.

A deduplication check only ever adds: each new id creates an entry and nothing deletes one, so the retained set grows with distinct ids for the life of the job. Only a removal rule bounds it.

The running total and the repeat-identifier dropper hold entries between records: one accumulator per user, one entry per identifier already seen. The filter holds nothing, because its verdict depends only on the record in hand.

The cap is the memory of that one worker process, shared with everything else the process is doing. There is normally no overflow path, so passing it degrades and then kills the worker rather than quietly moving entries to disk.

An identity label written with the entries. Either the author pinned a stable step identifier — a name fixed to that stateful step — or the runtime derived one from the step's position in the job's step graph, which later edits can shift.

Exactly one worker handles every record for a given key, so its entry has a single writer and needs no lock, no network call and no consistency protocol to read. The upstream redistribution is what gets records there.

Last-write expiry gives an entry a fixed lifetime from its last store or update; last-access expiry restarts the clock on reads too, so hot entries never leave. Deduplication wants last-write: a read there means a duplicate arrived.

Distinct live keys, bytes per entry and per-entry overhead set the size, with the retention horizon deciding which keys count as live. Input rate never enters for accumulators or deduplication entries, so a slow job can retain hundreds of gigabytes.

The ceiling moves from one worker process's memory to its local disk, and in exchange every read and write pays an encode and a decode, a disk read whenever the cache misses, and the background merging of the store's own files.

Additive changes usually survive: a field added with a default supplied for entries that lack it, and a widened type. Renames, a changed grouping key, a swapped record encoder and removed or reordered stateful steps do not, because the stored bytes no longer answer what the new code asks.

That is per-worker state: attached to one parallel instance of a step, not to a key. With no bucket to move, it is redistributed by a rule the author picks — the collected entries split across new instances, or every instance given the whole set.

The callback is a wake-up registered against one key and one moment; what bounds the retained set is the handler that deletes the entry when it fires. Pending callbacks are stored entries too, so one per record leaks just as badly.

Three routes: rewrite the stored snapshot offline into the new key and start from it; run a transition window where the new build writes the new key and still reads the old; or start empty and rebuild from the source. Each trades downtime, complexity or a period of wrong numbers.

The key space is cut into a fixed number of buckets at the job's first start, and a key never changes bucket. A width change therefore moves whole buckets between workers, and can never use more workers than there are buckets.

Every removal rule trades correctness for survival: dropped deduplication entries readmit duplicates, a dropped join side silently loses matches, a cleared activity entry splits one burst in two. It buys a set that stops driving snapshot size and restart time upward.

Restart time tracks the bytes a job reads back before its first record, and the retained set is those bytes. Latency rises because the working set outgrows the memory the store had for it. Both curves follow one growing entry count.

It missed records buffered awaiting their group, both sides of a join held until a match is possible, deduplication entries, pending per-key wake-ups, and the per-entry overhead the holder adds. Each is counted on a different quantity from accumulators.

Forty gigabytes cannot live as objects in a sixteen-gigabyte process, so a disk-backed store with a memory cache fits, and the two gigabytes touched per cycle are mostly cache hits. Restore time or an unstable hot set can overturn it.

Require a written retained-set budget per job before first deployment: entry categories, live key cardinality with its source, the horizon per category, bytes per entry, and the product. Fix the units so numbers compare, and require re-declaration when the grouping key or horizon changes.

Cost is paid per cycle, not per access: each cycle writes a file per state partition to durable shared storage, plus periodic full copies, so partitions multiplied by cycle rate sets the bill.