Machines for the Job

On pieces of work that have not started yet, normally from the next round of work onward. A piece already running on another worker process is not cut in half and shared, and finished work is not redone to spread it more evenly.

Everything that machine held goes at once: every worker process on it, the pieces they were running, anything cached there, and intermediate output produced there that other workers had not yet fetched. Finished work can therefore have to be produced again.

Work slots in one worker process share that process's single memory pool, its local scratch space, its fixed start-up overhead and its fate: they compete for the same memory, and all of them die together when the process does.

Nothing starts until machines free up. Submitted jobs wait in an admission queue, and a share policy — strict order of arrival, weighted shares, or a guaranteed minimum per group — decides which waiting job the freed capacity goes to.

One process, the coordinating process, turns the program into pieces of work, hands them to worker processes, tracks which finished, and receives anything the program asks to bring back. It runs no piece of the data itself.

A job over endless input keeps per-key state, and which worker process owns a given key follows how many worker processes there are. Change that count and the retained state has to move to its new owners before any record can be processed correctly.

Fewer, larger worker processes pay the fixed per-process overhead fewer times, keep more handoffs inside one address space and give any single unit of work a bigger pool to draw on. They also lose more when one dies and are harder to place.

The coordinating process runs out of memory. Every returned record leaves the workers and lands in that one process's heap on one machine, so the cost follows the result size, not the cluster size — adding worker processes does not help.

Only what was written to storage outside those machines. Caches held in worker-process memory, intermediate output written to local disk, temporary directories and logs left on the machines all disappear with them, so the next run starts cold.

Mostly yes. Two worker processes have separate address spaces, so records are encoded into bytes at one end and decoded at the other even on one machine. Sharing a machine skips the wire, not the encoding — only slots inside one process avoid it.

Under strict order-of-arrival admission the overnight job holds the pool and the queue is blocked at its head, so short work inherits the long job's runtime. Weighted shares, a per-group guaranteed minimum, a ceiling on one job's share, or reclaiming capacity all break it.

Inside the cluster, the coordinating process sits close to the workers and outlives the submitting machine. Outside it, the session stays interactive but every scheduling message crosses a slower link, and closing the submitting machine ends the job.

It isolates everything that lived on the machines: memory exhaustion, a crashed coordinating process, runtime and library versions, restarts and upgrades. It does not isolate shared storage, a shared metadata service, the quota with whoever grants machines, or the credentials both teams use.

A worker process with no piece in flight can still be holding output that other worker processes have not fetched yet. Release it and that output goes with it, so the pieces that produced it may have to run again — often costing more than the release saved.

Because that machine held intermediate output from earlier rounds that later pieces were still fetching. Losing it turns completed pieces back into work to be done, and the later the withdrawal, the more such output has accumulated.

Everything that lived in that process goes: all eight units of work in flight, from the beginning rather than from where they stopped, plus any intermediate output and cached data held only there. The process's size is the multiplier on all of it.

It loses the worker processes taken, every piece of work they were part-way through, and anything they still held that other workers had not yet consumed. The job itself survives; only what lived on the reclaimed capacity has to be produced again.

Losing a worker process costs the pieces it was running and the intermediate output it still held; the rest of the job continues. Losing the coordinating process leaves nothing planning, handing out or tracking work, so the job stops.

Place by wait tolerance and lifecycle ownership, not by team. Short interactive work needs machines already up; long scheduled jobs earn their own cluster and their own runtime versions; work with no operator behind it fits a service that shows no machine. Then limit how many models you operate.

Every change stops the job while its retained state is relocated, and that cost follows how much the job holds rather than how much capacity moves. A policy that reacts faster than the pause is long spends its savings on pauses and provokes further changes.

For jobs that redo more than they save: long runs whose later rounds depend on output held only on those machines, and latency-bound continuous jobs, where each withdrawal buys a pause, a rewind and a backlog that a job with no spare throughput cannot clear.

It waits forever. Under all-or-nothing placement a job starts only when every worker process it asked for is available together, so a request above the ceiling is not slow but impossible, and no amount of waiting changes it.