Your platform's default is one submission per pipeline - which boundaries should become scheduler edges instead, and what does each one cost?

## Why the default is the right default Inside one submission - one whole program handed to the cluster and run end to end - an edge carries records and the engine owns the handoff. Nobody names the intermediate, nobody owns its schema, nobody pays for it after the run, and on engines that rewrite a declarative program into a physical plan the two sides can be reorganised together. In the workflow graph, an edge means only that the downstream step starts after the upstream one reported success, and **nothing travels along it** - so every promoted boundary converts a free, invisible handoff into a durable artefact somebody has to design. That asymmetry is why 'one submission per pipeline' is a sound default and why each departure from it should be argued. ## The four arguments that actually hold 1. **Two genuinely different programs.** Different languages, different runtimes, one of them a declarative statement handed to a service that owns its own storage, plan and capacity. No single submission can contain both, so the boundary exists whether you like it or not; the only question is what crosses it. 2. **The intermediate has consumers.** If other pipelines, other teams or an analyst already read that dataset, you were materialising it regardless. The write is not a new cost, so the boundary is nearly free - it costs one start-up. 3. **The downstream waits on something that is not a record.** A file another organisation delivers, an approval, a signal from a system outside the pipeline. Waiting is precisely what an ordering edge is for, and jamming it inside a submission means a long-running program doing nothing but holding capacity. 4. **The two halves want different capacity.** A short, wide stage and a long, narrow one inside one submission usually hold one allocation shaped for the wider of the two. Splitting lets each ask for what it needs. This is a resources argument and it is real, but it is the weakest of the four, because the saving has to beat the materialisation. ## What each promoted boundary costs - **A named artefact.** A location, a format, a schema somebody owns, a rule for what a rerun does to it, and a deletion policy. This is the cost that outlives the decision. - **A full write and a full read**, unless argument 2 applied and you were writing it anyway. - **A second start-up**, whose size depends on how the machines are supplied. - **No rewriting across the boundary** on engines that would have done so. Concede the variation honestly: an engine that executes per-record functions almost literally, and the oldest disk-to-disk designs, were not going to rewrite much across it in the first place. - **A second picture.** The pipeline's shape now lives in two systems, and a reader has to hold both to know what runs. Two pictures is a real and permanent tax on comprehension. ## Arguments that should not decide it | the argument | why it fails | |---|---| | 'the workflow graph is where the team looks' | legibility of a diagram, paid for with a materialisation per arrow; fix the visibility, not the architecture | | 'the scheduler gives us failure handling and history' | what a scheduler does about a failed step is a real subject, but it is a property of that system rather than a property of *this* boundary - it argues for a scheduler, not for cutting here | | 'splitting makes it faster' | each split adds a write, a read and a start-up; the only speed argument is the capacity one, and it must beat that bill | | 'the intermediate is big' | size argues for fewer boundaries, not more | ## The two failure modes of a platform standard **Over-promotion** is the common one: the workflow graph becomes the place where all structure is expressed, so stages of one computation become steps, data-sized fan-out migrates there, and the pipeline degenerates into a launcher loop with storage between every arrow. The symptom is a graph whose box count tracks the data. **Under-promotion** is rarer and worth guarding against so the standard is honest: one submission containing several teams' unrelated work, where a dataset many people need is invisible inside it, nothing can be run on its own, and the blast radius of any change is the whole thing. A workable standard says three things: the default is one submission per pipeline; a boundary is promoted only on one of the four arguments above, recorded in one line where the pipeline is defined; and every promoted boundary's artefact has a named owner, a schema and a deletion rule from the day it is created. What makes it stick is the last clause - a boundary with no owner for its intermediate is a boundary nobody decided on.