Should the coordinating process run inside the cluster beside the workers or on the machine that submitted the job?
answer
- distance and lifetime, the two axes
- control messages per piece per round
- job dies with the submitting machine
- returned memory moves, does not shrink
- no choice on a managed service
basics
~20 sInside the cluster, the coordinating process sits close to the workers and outlives the submitting machine. Outside it, the session stays interactive but every scheduling message crosses a slower link, and closing the submitting machine ends the job.
solid answer
~50 sCoordinator placement is whether the coordinating process — the one process that plans the pieces, hands them out and tracks what finished — runs inside the cluster beside the worker processes or outside it on the machine that submitted the job. Two things move with it. **Distance:** the process exchanges several control messages per piece per round of work, so with tens of thousands of pieces a link measured in tens of milliseconds costs real elapsed time, and anything the program brings back crosses it too. **Lifetime:** outside the cluster, the job lives exactly as long as the submitting machine and its network, so a closed laptop is a dead job. Scheduled production work therefore belongs inside; interactive and exploratory work is what outside placement is for. Not every supply model offers the choice — a managed compute service, where you are shown no machine, owns it for you.
go deeper
Recall the two placements and the one consequence that always applies: if the coordinating process runs on the machine that submitted the job, the job ends when that machine goes away.
Explain the latency side properly — several control messages per piece per round of work, paid at the link's latency — and separate it cleanly from the memory question, which placement relocates rather than reduces.
Recognise the symptom in production: idle workers, a very large piece count and a distant coordinating process means waiting on round trips, not missing capacity, and the repair is placement or piece size rather than more machines.
The policy angle is standardisation: letting individuals submit scheduled work from their own machines gives the platform a job lifetime it cannot see or control, and the cost of forbidding it is only the convenience of interactive runs, which can be granted separately.
## What the choice is Every distributed job has one coordinating process: the single process that plans the pieces, hands them out, tracks what finished, and receives anything the program asks to bring back to one place. **Coordinator placement** is the question of where that process runs — inside the cluster, on one of the same machines that host the worker processes, or outside it, on the machine that submitted the job. The worker processes are inside the cluster either way. Only this one process moves, and it is worth being precise about what moves with it. ## What crosses the link 1. **Control traffic, which scales with pieces multiplied by rounds.** Handing out a piece, reporting its completion, re-issuing a failed one and the periodic liveness exchanges are each a small message, and a job cut into tens of thousands of pieces per round of work generates them by the hundred thousand. Each is tiny; the round trips are what cost, and they are paid at the link's latency rather than its bandwidth. 2. **Anything the program brings back.** Returned records cross to wherever that process sits. Placement does not change how much memory they need — that is the same figure in either location — but it changes which machine's memory it is, and a submitting laptop is usually the smaller of the two. 3. **The program's own ordinary code.** The parts of the submitted program that are not distributed operations execute in this process, so their dependencies, their configuration and their access to any external system are resolved wherever it runs. ## The two placements side by side | property | inside the cluster | on the submitting machine | |---|---|---| | latency of control traffic | machine-to-machine within one network, negligible per message | crosses whatever link the submitter has, and multiplies by piece count | | job lifetime | independent of the submitter; disconnecting does not end the run | exactly the submitting machine's and its network's; closing it ends the job | | memory for returned records | a cluster machine's, sized with the cluster | the submitting machine's, usually much smaller | | interactive use | awkward: the program's output is inside the cluster | natural: results and standard output are local and immediate | | reaching external systems the program needs | needs the cluster network to reach them | has whatever the submitting machine has | | operational visibility | the run's records live with the cluster's | the run's records live on a machine that may not be collected from | ## Which to choose - **Scheduled or long-running production work: inside.** It is the only placement where the job's life does not depend on a machine nobody is watching, and the only one where a large piece count does not pay a latency tax on every handout. - **Interactive, exploratory or development work: outside.** Immediate output, a local session, simple iteration; the jobs are small and short, so both the control traffic and the exposure are small too. - **A managed compute service — a service you hand work to and are never shown a machine by — offers no choice at all.** The provider owns the coordinating process along with everything else, and the equivalent lever is how much you ask to have returned. Some engines and some supply models likewise offer only one placement, so it is worth checking rather than assuming the knob exists. ## What the choice does not change It does not change how much memory bringing results back to one place needs, only whose memory. It does not change the plan: planning does not depend on the planning machine's hardware. It does not change where the data goes — records moving between workers never route through the coordinating process under either placement. And it does not change which paths the workers can read: input locations are resolved against the cluster's view of the world, not the submitter's, which is why a job that reads a file only the submitting machine can see fails under both placements. ## The symptom to recognise A job whose worker processes are visibly idle much of the time, whose piece count is very large, and whose coordinating process sits on a distant machine, is usually not short of capacity: it is waiting on round trips. Moving that process inside the cluster, or cutting the input into fewer and larger pieces, are the two directions of repair — and choosing between them is a different subject from this one.
- A job with 50,000 pieces per round runs far slower with the coordinating process outside the cluster. What is happening?Round trips. Each piece costs a few small control messages — handing it out, reporting completion, sometimes re-issuing it — and fifty thousand pieces per round turn a link's latency into minutes of elapsed time per round while the worker processes wait. Bandwidth is not the constraint; the number of exchanges multiplied by the distance is.
- Does moving the coordinating process into the cluster fix an out-of-memory error when results are brought back?Only by accident. The amount of memory needed is the size of the returned result and does not change with placement; what changes is which machine must supply it, and a cluster machine is often larger than a laptop. If the result is genuinely large the same failure follows the process inside, and the real repair is to stop returning the rows.
saying these in an interview costs you the question
- Thinks placement only decides where the log output appears
- Runs scheduled production work with the coordinating process on a personal machine
- Assumes every supply model offers a choice of placement
- Believes moving it inside reduces the memory returned results need
- Ignores that control traffic grows with pieces times rounds
- Expects input paths to resolve against the submitting machine