A worker fed by two upstream inputs receives the saving marker on one input long before the other. What are its options, and what does each cost?

## Why one worker sees the marker twice, at different times A capture begins at the workers reading the input, and each of them injects its own copy of the **marker travelling with the records** — the special element that each worker passes along after saving its own part. A worker in the middle of the graph fed by two upstream paths therefore receives two copies of the same marker, and they will not arrive together. The paths differ in the number of steps, in how much is buffered along them, and in how fast each source is delivering; an input that is behind and catching up delivers its marker late by roughly the amount it is behind. A worker with a single input never meets this. It saves when its one marker arrives and forwards it. The problem is purely one of fan-in: joins, unions, and anywhere a redistribution brings several producers into one consumer. ## Option one: hold back the input that arrived first The worker stops consuming from the input that has already delivered its marker, buffering whatever continues to arrive there, and keeps processing the other input until its marker arrives too. Then it saves its part and forwards the marker. - The saved part is a **clean prefix on both inputs** and as small as it can be, because nothing in flight has to be written down. - The cost is a genuine pause on the early path, which turns into pressure back up that path as producers find their consumer no longer draining them. - The pause lasts as long as the gap between the two paths, so it is small when the job is healthy and large exactly when one input is backlogged. ## Option two: save at once and write down what is in flight The worker saves as soon as the first marker arrives, and from then until the second marker arrives it records the incoming records from the other input **into the picture** as well as processing them. - Nothing pauses, so latency does not spike and nothing pushes back upstream. - The picture is larger by those buffered records, so it takes longer to write and costs more bandwidth. - On resuming, the recorded records have to be pushed through the worker again, in order, before any fresh input, which makes the restore path slower and slightly more intricate. ## Side by side | | Hold back the first input | Save at once, record what is in flight | |---|---|---| | Pause during capture | Yes, on the early path | None | | Size of the picture | Smallest possible | Larger by the buffered records | | Resuming | Load the parts and go | Load, then drain the recorded records first | | Worst when | One input is backlogged or skewed | Buffers between steps are large | | Best when | Inputs are balanced and the job keeps up | The job is catching up and must not be slowed | ## What varies between runtimes Not every runtime offers both, and where both exist the default differs; some switch from the first form to the second only when a capture is running late, and some offer only the waiting form. Neither question arises in a runtime built on **repeated small finite jobs** — serving an endless input by cutting it into short bounded pieces and running a complete little job over each — because the seam between two pieces already has nothing in flight and no marker has to travel. The **two-phase disk-to-disk model**, which materialises each phase's output to durable files before the next phase reads them, does not face it either. ## What the operator actually sees - Capture duration rising, and rising because of one or two workers rather than all of them. - Added latency on one input path only, which is the wait rather than the write. - Captures starting to overlap, or being abandoned, once the wait approaches the gap between captures. ## The judgement the question is really after Waiting is close to free on a healthy job with balanced inputs, and gives the smallest and cheapest picture. It is at its most expensive when one input is behind — which is precisely when you most want captures to keep completing, because that is when a restart would hurt most. Recognising that inversion, rather than naming a preferred setting, is what separates a candidate who has operated a job of this shape from one who has read about it.