Engine or Orchestrator

## Two pictures that share a word A **workflow scheduler** and a **cluster execution engine** both get drawn as boxes and arrows, and both graphs are acyclic: a set of steps with ordering edges and no cycles, a directed acyclic graph. The shape is genuinely the same. The shape is also the only thing that is the same, because the word says the arrows never loop back and says nothing at all about what an arrow *is*. Spelling the two systems out: - a **workflow scheduler** is a system whose only job is to start other programs in a required order and record whether each one finished; it moves no records itself; - a **cluster execution engine** is a system you hand a whole program to, which splits that program's work across many machines, runs the pieces and puts the results back together. Handing it that program - packaged with the resources it is asking for - is one *submission*, and the engine runs that submission end to end. ## The scheduler's edge means 'after' An edge from step A to step B in the workflow graph is a statement about *when*: do not start B until A has reported success. That is the whole of its content. Nothing travels along the arrow. Each step is usually its own process, started and then watched - sometimes it is a unit run inside a shared worker instead, but either way it has its own memory and that memory is gone when it exits. So the consequence candidates miss is immediate: **if B needs anything A produced, A must write it somewhere B can independently find, and B must be told where.** The scheduler supplies neither the storage nor the name. A table, a path in the shared storage every machine can read, a message somewhere - a real durable location, chosen, named and eventually cleaned up by you. ## The engine's edge carries records Inside one submission that the engine runs end to end, an edge between two steps means the records the upstream step produces become the input of the downstream step. The engine owns that handoff, and where it physically happens is its choice rather than yours - and this is exactly where real engines differ from each other: - some keep the handoff in memory and run adjacent steps as a single pass over the data; - some write it to a file on the producing worker's own local disk and have the consuming side fetch it across the network; - the oldest disk-to-disk designs in this family write each round's whole output to durable shared storage before the next round reads it, so a chain of rounds pays a full write and a full read between every pair. What is constant is not the mechanism but the **ownership**: you never name the intermediate, never create it, never delete it, and nothing outside the submission can see it. A claim like 'the engine never writes intermediates' is true of one design and false of another; a claim like 'the engine chooses where the intermediate lives' is true of all of them. ## Side by side | | edge in the workflow graph | edge inside one submission | |---|---|---| | what it asserts | B starts after A succeeded | records flow from A into B | | what travels along it | nothing | the records themselves | | the intermediate | you choose, name and delete it | the engine chooses; it is unnamed | | what sits at each end | a whole program | an operator inside one run, which starts no process of its own | | who can see it | anything that can read the location | nothing outside the submission | | what crossing costs | materialising the data, plus a second start-up | whatever handoff the engine picked | ## The test that separates them Put one question to every arrow in your picture: **if I deleted this edge, would records be lost, or only ordering?** 1. Records would be lost - the arrow is carrying data, so it belongs inside one submission where an engine can own the handoff. 2. Only ordering would be lost, and the two sides are separate programs - the arrow belongs in the workflow graph. 3. Both, because the downstream step reads a file the upstream step wrote - you have a scheduler edge with a data dependency bolted to the side of it, which is legitimate when that file is a real product other things read, and an accident when it exists only because the picture was drawn in the wrong system. ## Why the confusion is expensive The two mistakes are mirror images. Treat an engine edge as a scheduler edge and every intermediate becomes a named, written, read-back dataset with a start-up between the halves. Treat a scheduler edge as an engine edge and you get one enormous submission containing work that has no data relationship at all, where the boundaries that mattered to people are invisible. Neither is a scale problem and neither is fixed by a bigger cluster: they are both the same category error about what an arrow means.