Why does a worker process allocating an object per record show repeated pauses rather than a memory error?

## What is actually happening A **worker** is one operating-system process on one machine that runs some of the job's work and owns a fixed amount of memory nothing else can borrow. Several **units of work** - each one worker's share of one step, scheduled and retried on its own - typically run inside it at once and share that memory. Where the worker's language runtime reclaims memory automatically, it periodically finds objects that nothing refers to any more and frees them, and while it does that it stops the worker's own work. That is **automatic reclamation**, and the stop is **a reclamation pause**. It is not an error condition. It is the normal price of a runtime that frees memory for you. Now consider a step that holds each record as **host-runtime objects** - ordinary objects of the worker's language, one per record plus one per field that is itself an object. Every record that flows through the step creates a small cloud of objects, uses them for microseconds, and abandons them. A worker handling millions of records per minute is therefore creating and abandoning tens of millions of objects per minute. Each one is individually trivial. In aggregate they are the worker's dominant activity. ## Why it is time, not space The reclamation is succeeding. Objects abandoned immediately are the cheapest kind for most runtimes to reclaim, so nothing accumulates and no allocation is refused. What is consumed is the worker's own wall-clock time. Two things make it worse than the raw volume suggests: - **Survivors cost more than short-lived objects.** If the step holds objects rather than dropping them - accumulating a grouping table, buffering for a sort, building one side of a join - those objects live long enough to be copied or promoted by the runtime instead of being discarded cheaply. - **A larger memory budget is not automatically a shorter pause.** More room can mean fewer passes but a bigger space to walk, so the pauses come less often and last longer. Raising memory is not a reliable remedy for this symptom. ## Reading it from the run's own numbers This is settled by the run's reported numbers - whatever the engine reports per unit of work after a run - not by argument. The signature is: 1. **High wall time, low useful processor time** for the affected units of work. 2. **No bytes written to local scratch disk** - nothing overflowed and wrote part of its working set out, which is the other common cause of a slow unit. 3. **No report of a refused allocation** and no process killed at the ceiling the platform enforces. 4. **Whatever the runtime reports as time spent reclaiming**, if the worker exposes it, which turns the diagnosis from an inference into a measurement. The near neighbours are worth excluding explicitly. A unit of work that simply received far more records than its neighbours is an uneven-work problem and belongs to a different subject; the tell is that its useful processor time is high, not low. An operator that could not hold its working set and wrote part of it to disk attached to the worker - spilling - reports the bytes it wrote. Neither looks like this one once you have the per-unit numbers in front of you. ## What actually reduces it 1. **Allocate fewer objects per record.** Reuse a mutable container across records instead of constructing a fresh one; avoid wrapping each value in its own object; avoid building an intermediate collection per record just to fold it immediately. 2. **Let the engine keep records packed where it offers that.** **Engine-managed binary records** - records kept as a packed byte layout the engine allocates and interprets itself, fields read at known offsets - mean a step can compare, hash and copy records without materialising objects at all, and the churn disappears with them. 3. **Work in batches.** One call handling many records allocates once for the batch instead of once per record. 4. **Cut the records down before the expensive step.** Fewer records and fewer fields crossing into object form is a linear reduction in the churn. Raising worker memory is not on this list, and "add more machines" is not either: the allocation rate per record is unchanged, so the same churn is simply spread wider. ## Where this varies Not every worker runtime reclaims automatically, and not every engine holds records as objects. On a runtime that frees memory by counting references or by explicit management, the same allocation volume still costs - it shows up as time in the allocator and in freeing, spread evenly rather than concentrated in pauses. On an engine that carries records packed through its own operators, a pipeline written entirely in those operators barely allocates per record at all, and this whole failure mode only appears where the pipeline crosses back into host-language code. Establish which of those you are on before predicting the symptom.