When two versions of a job's logic run side by side over the same input and only one publishes, which effects of the silent arm still reach the outside world?

## Not publishing is something you build A side-by-side run means both versions of the transformation logic execute over the same input at the same time, with only one of them reaching readers. The phrase *only one publishes* describes an outcome, not a switch. Publishing is not a single act on most platforms: it is a write to a destination, plus whatever the write causes, plus whatever the logic does on its own along the way. Each of those is a separate thing to contain, and the ones that get missed are never the destination. What counts as publishing also varies by platform: committing a set of files to shared storage, updating rows in a store that something queries, or emitting records that a downstream reader consumes. Contain all three shapes you actually have, not the one you happen to be picturing. ## The paths by which the silent arm still reaches the world 1. **The destination.** The obvious one. Give the candidate arm its own output location and make sure nothing defaults the writer back to the real one. It must, however, keep the same write *mechanism* and the same output shape, or you end up comparing writers instead of logic. 2. **The input reader's recorded position.** If both arms read through a shared identity that records how far the input has been consumed, the silent arm can mark records as read and the publishing arm will never see them. Give each arm its own reader identity, or feed the candidate from a stored copy of the same records. The mechanics of how a log or queue records consumer progress are a messaging subject, not this one; what matters here is that the identity must not be shared. 3. **Side effects inside the logic.** An **opaque step** is a hand-written per-record function the engine can only call, never inspect or rewrite. If one of them posts to an external service, sends a message, or writes an audit row, it fires twice — once per arm — and nothing in the job's step graph shows it, because the engine cannot see inside the function. Route these to a discard target in the candidate arm, or make the arm identity an input the function honours. 4. **Downstream triggers.** A consumer that reacts to a new file appearing, or to a row changing, will react to the candidate arm's output if it lands anywhere watched. Being in a different folder under the same watched prefix is not isolation. 5. **Telemetry and alerting.** Failures, lag and throughput from the candidate arm land on the same dashboards and page the same on-call unless every signal carries which arm emitted it. A noisy candidate arm that wakes people at night gets switched off before it has told you anything. 6. **Shared capacity and the source.** Two arms roughly double the compute and read the source twice. On a shared pool, the candidate can starve the publishing arm; on a rate-limited source, it can throttle it. The publishing arm keeps priority, always. | leak path | what you see | containment | |---|---|---| | shared destination | readers get the candidate's numbers | separate location, no default back to the real one | | shared reader identity | published output silently misses records | separate identity, or a stored copy of the input | | side-effecting per-record function | messages and rows written twice | discard target, or arm identity passed into the function | | watched output location | downstream jobs run on the candidate's output | write outside every watched prefix | | untagged telemetry | the on-call is paged for a job nobody depends on | tag every signal with the arm, route candidate alerts elsewhere | | shared pool or rate-limited source | the publishing arm slows or throttles | priority to the publishing arm, cap the candidate's width | ## The distortion that is not a leak: warm-up A long-lived candidate arm starts with nothing in its **retained set** — everything a running job holds between records, such as counters, buffered join sides and the last value per key. Its earliest outputs are therefore different from the publishing arm's for a reason that has nothing to do with the logic change, and reading those rows as a real difference wastes a day. This varies sharply by runtime model: where continuous work runs as a rapid succession of small finite runs, each run is bounded and the distortion is short; where the runtime is record-at-a-time with a key-bound retained set, the arm may take a long time to become comparable; and a periodic job over a finite input has no warm-up at all. Either discard the warm-up window from the comparison or start the candidate arm early enough that it is settled before the window you intend to read. ## What must stay identical The candidate arm should differ from the publishing arm in exactly three respects and no others: - **its destination**, which must sit outside every watched location - **its reader identity**, so it cannot consume on the publishing arm's behalf - **its telemetry tag**, so its failures and lag are attributable and routed away from the on-call Everything else stays the same — same engine, same input, same period, same write mechanism, same output shape, and ideally the same width. Every further difference is a difference the comparison will faithfully report and you will then have to explain, and each one you allow costs you a day of reading rows that were never about the change.