Why should an orchestrator trigger transformation work on external compute rather than run it itself?

## Control plane and data plane An orchestrator is a **control plane**. It holds the dependency graph, decides what is eligible to run, enforces concurrency limits, retries what failed, and records what happened. The systems that hold and process data — the warehouse, the processing cluster, the streaming engine — are the **data plane**. Keeping those roles separate is the single most consequential design habit in pipeline engineering, and violating it is the most common way a working pipeline becomes an unworkable one. The violation always looks reasonable at the time. A task reads a query result into memory, loops over it, and writes rows back. It works on a laptop against a small table. It works in production for six months. Then the table grows. ## What breaks, in the order you meet it **Memory becomes the ceiling.** The largest dataset the pipeline can process is now whatever fits in one worker process. That limit has nothing to do with the capability of the warehouse holding the data, and it moves only when someone pays for bigger scheduler infrastructure — infrastructure that sits mostly idle, because coordination needs almost nothing. **Failures spread sideways.** Orchestrator workers typically run several tasks concurrently. A task that allocates too much triggers the operating system's out-of-memory killer, which takes down the process and every unrelated task sharing it. A slow database load in one pipeline kills a report in another. The blast radius of a data problem becomes the whole scheduler. **The target's parallelism is wasted.** A warehouse asked to run an aggregation splits it across many workers and reads only the columns needed. The same logic implemented as a Python loop over fetched rows runs on one core, serially, having transferred every column across the network first. It is slower by orders of magnitude at exactly the workload the warehouse is best at. **Retries become expensive and unsafe.** If the work lives in the worker, a retry re-reads and re-processes everything from the start, and any rows already written may be written again. If the work lives in the target, the retry is one statement — and a statement that overwrites a partition or merges on a key is naturally idempotent. **Operations get coupled to data volume.** Orchestrator upgrades, worker restarts, node rotations and deployments all become risky, because they can kill hours of in-flight processing. A control plane should be restartable at any moment; one that holds work in progress is not. **Observability degrades.** When the warehouse executes the statement, its query history holds the plan, the bytes scanned, the duration and the cost, and the orchestrator holds the run record and the dependency edges. Each system reports what it knows. When the worker does the work, the only artefact is a task log, and diagnosing slowness means adding print statements. ## The pattern that scales: submit and poll A well-behaved transformation task does four things: submit an instruction to the system that owns the compute; record the identifier it returns; poll or wait for terminal status; translate that status into task success or failure, carrying back the error and a link to the external run. No rows pass through the worker. The task's memory usage is constant whether the table has a thousand rows or a trillion. This also fixes the semantics of a timeout. Cancelling a task that holds the work in-process leaves the target's state unknown; cancelling a submit-and-poll task can explicitly cancel the remote job, or deliberately leave it running and reattach on the next attempt. ## The legitimate exceptions This is a rule about volume, not about doing nothing. Perfectly reasonable work inside a worker: issuing SQL and waiting; calling an API to start a job; reading a manifest or a schema; passing small metadata such as a partition key or a row count between tasks; sending a notification; evaluating a branch condition. The heuristic is that anything crossing the worker should be small, bounded and known — configuration and control signals, not payload. The boundary case worth naming is the small file. Moving a few megabytes through a worker is fine and often simplest. What is not fine is a design whose correctness depends on the payload staying small, with no check and no alternative when it does not. If the volume is unbounded by nature, route it through a system built for volume from the start. ## Why interviewers ask this Because it separates people who have operated a scheduler from people who have only written tasks. The candidate who has been paged because one report's memory spike killed a night's worth of unrelated pipelines never designs it that way again — and can explain, concretely, what the orchestrator should have been doing instead.