Sixteen data-parallel replicas deliver six times one device's throughput - do you buy more devices?
answer
- Thirty-seven percent efficiency, not sixteen times
- Decompose the step before spending
- Communication bytes track parameters, not batch
- Free levers before hardware levers
- Throughput is not time to quality
basics
~20 sNot before you know where the missing throughput went. At 37 percent scaling efficiency each added device returns about a third of a device, and a step bound by gradient communication only gets worse with more replicas.
solid answer
~50 sSixteen for six is roughly 37 percent efficiency - each added device is paying for itself about a third of the time. The decision is not about hardware until the step-time breakdown is known: per-device compute, exposed gradient communication, input-pipeline wait, and straggler spread. Communication is the usual culprit: a ring reduction moves about twice the gradient bytes per device regardless of replica count, so the ratio that matters is parameter count against work per device. A 60M-parameter convnet at a healthy micro-batch hides its reduction almost entirely; a 1.3B-parameter text model at a micro-batch of two can spend longer reducing than in its own backward pass. Cheap fixes come first: raise the per-device micro-batch, overlap the reduction with the backward, reduce gradients in lower precision. Buy devices only where the measured curve is still near-linear, and judge by time to target quality rather than samples per second.
go deeper
Know that adding devices does not multiply throughput one-for-one, and that the gap comes from time spent communicating gradients or waiting for data rather than from the devices being slow.
Explain the ratio: gradient bytes are set by parameter count while compute per step is set by the micro-batch, so raising the per-device batch is the standard way to make the reduction a smaller share of the step.
Show the diagnosis you would run - ablating the reduction, feeding synthetic batches, checking per-replica step-time spread - and the order in which you would apply the free fixes before asking for hardware.
Own the decision framing: refuse to answer from one efficiency number, price the marginal device against interconnect and against alternative strategies, and state whether cost per sample or wall-clock to a deadline is the currency being optimized.
## Read the number first Sixteen devices returning six times one device's throughput is about 37 percent scaling efficiency. Framed as a purchase: the sixteenth device delivered a fraction of a device's worth of work, and the seventeenth will deliver less. Nobody should approve more hardware against that curve without knowing why it bent. ## Decompose the step Four candidates account for nearly all lost efficiency in data-parallel training: 1. **Exposed gradient communication.** The all-reduce that could not be hidden behind the backward pass. 2. **Input starvation.** The data pipeline cannot feed sixteen replicas as fast as it fed one; devices idle waiting for samples. 3. **Stragglers and imbalance.** A synchronous step is as slow as its slowest replica. Variable-length inputs, a hot neighbour on a shared machine, or one thermally throttled device sets the pace for all sixteen. 4. **Fixed per-step overhead.** Launch and synchronization costs that were amortized over a large single-device batch and are now amortized over a small per-device one. They are distinguishable by experiment. Run with the reduction removed (replicas training independently, results discarded) - if throughput jumps, communication is the binding constraint. Feed a synthetic in-memory batch - if throughput jumps, the input pipeline is. Log per-replica step times - a wide spread is stragglers. Raise the per-device micro-batch - if per-sample time falls sharply, fixed overheads and communication were being amortized badly. ## Why parameter count drives the communication cost Per step, each device must exchange its gradients. For a bandwidth-optimal ring reduction each device sends and receives roughly `2P` bytes for `P` bytes of gradient, essentially independent of the replica count - so adding replicas does not blow up per-device bytes, though it does add latency-bound stages. The compute per step, meanwhile, is proportional to the per-device micro-batch times the work per sample. So the ratio that decides everything is **gradient bytes against work per device**: - A 60M-parameter convnet at a comfortable micro-batch does a lot of arithmetic per sample relative to a small gradient. The reduction overlaps with the backward and near-linear scaling to eight or sixteen devices is routine. - A 1.3B-parameter model forced to a micro-batch of two because activations are enormous has a huge gradient and little compute per step to hide it behind. Such a run can genuinely spend more wall-clock in the reduction than in its own backward pass. This is why the same cluster gives one team beautiful scaling and another a flat curve: it is not the cluster, it is the ratio. ## The levers that cost no hardware - **Raise the per-device micro-batch.** More compute per step, unchanged communication - the single most effective lever, bounded by activation memory. - **Overlap the reduction with the backward pass.** Gradients arrive last-layer-first and can start reducing immediately; bundling small tensors into fewer messages amortizes per-message latency. - **Reduce in lower precision or compress.** Halving the bytes roughly halves the exposed communication time; the numerical cost has to be checked, not assumed. - **Fix the input pipeline.** Often the cheapest win of all, and frequently misdiagnosed as a communication problem. - **Kill the stragglers.** Balance shard sizes and input lengths; isolate the run from noisy co-tenants. Exhaust these before spending. It is common to recover most of the gap without touching the hardware order. ## Then decide, in economic terms If after the fixes sixteen devices give eleven or twelve times one device, the curve is worth extending and you can reason about where it bends next. If it stays at six, more devices buy less than a third of a device each and the money is better spent on interconnect, on a different parallelism strategy, or on not scaling at all. Two caveats belong in the decision: - **Throughput is not progress.** Samples per second is a proxy; the objective is time to target quality. Adding replicas grows the effective batch, and whether that enlarged batch still makes comparable progress per step depends on how the run is tuned - a separate question, but one that can quietly turn a throughput win into no win at all. - **Wall-clock sometimes justifies inefficiency.** Thirty-seven percent efficiency is a bad deal per sample and can still be the right call when a deadline, a research iteration loop, or a fixed launch date makes elapsed time the scarce resource. Efficiency is an economics argument, not a correctness one - state which currency you are optimizing. ## The answer they want A principal-level answer refuses the framing of the question. You do not decide on the strength of one ratio; you decompose the step, apply the free fixes, re-measure the curve, and then price the marginal device against the alternatives - in the currency the business actually cares about.
- How would you distinguish a communication-bound step from an input-starved one?Two cheap ablations. Run the replicas without the gradient reduction, accepting that the result is meaningless - if throughput jumps, communication was binding. Then run with a synthetic in-memory batch that never touches storage - if throughput jumps there instead, the input pipeline was. Per-replica step-time spread separates both from stragglers, where the slowest replica sets the pace.
- When is 37 percent scaling efficiency still the right decision?When elapsed time is scarcer than money. A launch date, a research loop where a two-day turnaround kills iteration, or a job that simply will not finish in the window on fewer devices. Efficiency is an economics argument: inefficient scaling that meets the deadline beats efficient scaling that misses it. Say which currency you are optimizing and why.
- What would you measure before approving more hardware?The scaling curve itself - the same job at one, two, four, eight and sixteen replicas - plus the fraction of step time that is exposed communication at sixteen and the per-replica step-time spread. A curve still near-linear at sixteen justifies extending it; one already flattening tells you the next devices return almost nothing and the money belongs elsewhere.
saying these in an interview costs you the question
- Assumes throughput scales linearly with replica count
- Orders hardware before measuring where step time goes
- Claims the all-reduce volume grows with batch size
- Believes a faster interconnect is always the fix
- Treats samples per second as time to target quality
- Never considers the input pipeline or stragglers