In tf.distribute, how do you choose between MirroredStrategy, MultiWorkerMirroredStrategy, and ParameterServerStrategy?
answer
- Climb the ladder only when forced
- One big box beats many small ones
- Synchronous means every replica alive
- Async buys preemption tolerance, costs freshness
- Replication never fixes a too-large model
basics
~20 sMatch the hardware and the failure model: MirroredStrategy for several GPUs in one box, MultiWorkerMirroredStrategy for several machines with a good interconnect, ParameterServerStrategy when workers are asynchronous or preemptible. All three replicate the model, so none of them fits a model too large for one device.
solid answer
~50 sStart with the smallest thing that works. `MirroredStrategy` covers one machine with several GPUs — synchronous all-reduce, no cluster configuration, and it is where most teams should stop, because a single 8-GPU host avoids every network problem. Move to `MultiWorkerMirroredStrategy` only when one host is genuinely insufficient: it is still synchronous, so it needs `TF_CONFIG`, gang scheduling, a fast interconnect, and it advances at the slowest worker's pace. `tf.distribute.experimental.ParameterServerStrategy` is the asynchronous option — workers push updates to parameter-server tasks through a coordinator, so slow or preempted workers do not block the rest, at the cost of stale gradients and a much harder debugging story. `TPUStrategy` is chosen by hardware, not preference, and brings its own constraints, notably static shapes. The decisive question is usually not throughput but the failure model: synchronous strategies need every replica alive at once.
go deeper
Know the shape of the menu: MirroredStrategy for several GPUs in one machine, MultiWorkerMirroredStrategy for several machines, TPUStrategy for TPUs, and that swapping between them is mostly a one-line change.
Explain what each buys and requires — synchronous all-reduce versus asynchronous parameter servers, TF_CONFIG for multi-worker — and that all of them replicate the whole model on every replica.
Argue from measurement: profile before distributing, quantify scaling efficiency across hosts, and weigh straggler sensitivity, interconnect bandwidth and restart-based recovery when recommending a jump.
Own the whole decision — capacity planning against one large host versus a fleet, whether the failure model tolerates gang scheduling, the reproducibility cost of asynchrony, and the porting budget a TPU move implies.
## Frame the decision correctly Candidates reach for the strategy that sounds most powerful. The useful framing is a ladder, where each rung buys scale and costs operational simplicity, and where you climb only when forced. ## Rung 0 — no strategy With no scope active, TensorFlow uses a default no-op strategy. One GPU, no coordination, nothing to debug. `tf.distribute.OneDeviceStrategy` is the explicit version, useful for testing that scoped code paths work before scaling. A large fraction of "we need distributed training" turns out to be an input pipeline that cannot keep one GPU fed — profile before you distribute, because adding replicas to an input-bound job buys nothing. ## Rung 1 — MirroredStrategy One machine, all its GPUs, synchronous all-reduce over NVLink or PCIe. No cluster config, no `TF_CONFIG`, no network partitions, one process to attach a debugger to. The intra-host interconnect is an order of magnitude better than anything between hosts, so an 8-GPU box scales far better than 8 single-GPU boxes. You can tune the reduction with `cross_device_ops`, choosing among implementations such as `tf.distribute.NcclAllReduce`, `tf.distribute.HierarchicalCopyAllReduce` and `tf.distribute.ReductionToOneDevice`, the last being mostly a debugging fallback. `tf.distribute.experimental.CentralStorageStrategy` is a variant that keeps variables on the CPU rather than mirroring them, occasionally useful when variables are huge relative to activations. **Stay here as long as you can.** The right question before climbing is "can we rent a bigger box?", and it very often ends the conversation. ## Rung 2 — MultiWorkerMirroredStrategy Same synchronous model, spread across machines, configured through `TF_CONFIG`. What you take on: - **Gang scheduling.** All workers must be up simultaneously; N-1 workers idle while the last is pending. - **Straggler sensitivity.** Every step ends in an all-reduce, so the slowest worker sets the pace. Heterogeneous hardware wastes the fast machines. - **Interconnect dependence.** Gradient exchange per step is proportional to model size; on commodity networking, communication can dominate and scaling efficiency collapses. Tuning `CommunicationOptions` toward NCCL helps on capable hardware but does not create bandwidth that is not there. - **All-or-nothing failure.** A dead worker stalls the collective. Recovery is restart-based, made affordable by `tf.keras.callbacks.BackupAndRestore`. The honest test: measure step time on 1 host and on N hosts. If N hosts give you well under N times the throughput, you have bought complexity, not speed. ## Rung 3 — ParameterServerStrategy `tf.distribute.experimental.ParameterServerStrategy` inverts the model. Variables live on dedicated parameter-server tasks; workers pull parameters, compute gradients, and push updates **asynchronously**, coordinated through `tf.distribute.experimental.coordinator.ClusterCoordinator`. Nothing waits for anything. What that buys: tolerance for preemptible or heterogeneous workers, elastic worker counts, and a natural fit for models whose variables are dominated by very large embedding tables that no single worker needs in full. What it costs: **stale gradients** — a worker's update is computed against parameters that have since moved, which changes convergence behavior and makes runs harder to reproduce — plus a more complex topology, more tasks to operate, and debugging that is genuinely harder because there is no global step boundary to reason from. The namespace itself signals maturity: it is still under `tf.distribute.experimental`. ## TPUStrategy — chosen by hardware `tf.distribute.TPUStrategy` is not an alternative you pick for architectural reasons; you pick it because you have TPUs. It requires connecting to the TPU cluster first — `tf.distribute.cluster_resolver.TPUClusterResolver`, `tf.config.experimental_connect_to_cluster`, and `tf.tpu.experimental.initialize_tpu_system` — and it inherits XLA's constraints, chiefly static shapes. Ragged inputs, dynamic batch sizes and data-dependent control flow that were fine on GPU need reworking. Budget porting time, not just cluster time. ## The constraint none of them lift Every strategy above is **data parallel**: each replica holds a full copy of the model. If the model plus its optimizer state does not fit on one device, adding replicas does not help — you get the same out-of-memory error on more devices. That is a different problem requiring the model to be sharded across devices, and it is not what the mirrored strategies do. Recognizing this distinction is the single clearest signal in this question; candidates who propose `MirroredStrategy` for a model that does not fit have not understood what replication means. ## The decision in one paragraph Profile first. Exhaust one host. Move to multi-worker only with an interconnect that justifies it and an orchestrator that can gang-schedule and restart. Reach for parameter servers when the fleet is asynchronous by nature or the variables are dominated by embeddings, and accept stale gradients as the price. Choose TPU when you have TPUs and can afford the static-shape port. And when the model itself does not fit, stop looking at `tf.distribute` strategies for the answer.
- A team asks for multi-worker training because one GPU is slow. What do you check first?Whether the GPU is actually saturated. Input-bound training is extremely common, and adding replicas to a job starved by its data pipeline multiplies cost for no throughput. Profile the step, look at device utilization and input latency, fix the pipeline, then scale within one host before crossing a network boundary.
- What does ParameterServerStrategy give up in exchange for asynchrony?Gradient freshness and reproducibility. A worker computes gradients against parameters that other workers have already moved on from, so updates are stale and convergence behaves differently from the synchronous equivalent. There is also no global step boundary, which makes debugging and exact run-to-run reproduction considerably harder.
- Why does moving from one 8-GPU host to eight 1-GPU hosts usually scale worse?Because every step ends in a gradient exchange sized by the model, and the intra-host interconnect is far faster than typical between-host networking. Synchronous training also waits on the slowest participant, so cross-host variance and stragglers now show up every step. Same replica count, much worse communication-to-compute ratio.
- A model plus its optimizer state does not fit on one GPU. Does any of these strategies solve it?No. All of them replicate the full model on every replica, so you hit the same out-of-memory condition on every device. That is a model-sharding problem, not a data-parallel one, and it is answered by sharding parameters and optimizer state across devices rather than by choosing a different mirrored strategy.
saying these in an interview costs you the question
- Reaching for multi-worker before saturating one host
- Proposing MirroredStrategy for a model that will not fit
- Treating cross-host scaling as linear in replica count
- Ignoring stale gradients when choosing parameter servers
- Picking TPUStrategy without budgeting for static shapes