skip to content

DDP & Process Groups

DistributedDataParallel replicates a model per rank and overlaps bucketed all-reduce with backward, while torchrun wires the process groups. Most multi-GPU interview war stories start here.

on this pageshow

questions

6

When torchrun launches a PyTorch DDP job, what does the training script read from the environment, and why must it call torch.cuda.set_device(LOCAL_RANK)?

level: juniorimportance: must knowfreq 32%

answer

  1. one process per GPU
  2. env:// is the default init method
  3. global rank vs rank within the node
  4. a CUDA process defaults to device 0
  5. the old launch module passed an argument

basics

~20 s

torchrun starts one process per GPU and exports RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT, which init_process_group reads through env://. Each process must pin itself to GPU LOCAL_RANK, or every rank on a node lands on GPU 0.

solid answer

~40 s

`torchrun --nproc-per-node=8` starts eight worker processes on each node and exports `RANK` (global, 0-15 on two nodes of eight), `LOCAL_RANK` (0-7 within the node), `WORLD_SIZE` (16), `LOCAL_WORLD_SIZE`, `MASTER_ADDR` and `MASTER_PORT`. `dist.init_process_group(backend="nccl")` assumes `init_method="env://"` when neither a URL nor a store is given, so it finds the rendezvous store and its own rank from those variables. `LOCAL_RANK` chooses the GPU: torchrun does not narrow `CUDA_VISIBLE_DEVICES` by default, so a process that never calls `torch.cuda.set_device(local_rank)` allocates on GPU 0 — eight replicas on one card, seven idle cards, and NCCL rejecting several ranks on one GPU. Then wrap with `DDP(model, device_ids=[local_rank])` and keep `RANK` for rank-0-only work such as logging and saving. In PyTorch 2.14, `python -m torch.distributed.launch` is deprecated in favour of `torchrun`.

code

python · 19 lines
python
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

local_rank = int(os.environ["LOCAL_RANK"])   # 0-7 on each node
torch.cuda.set_device(local_rank)            # before any CUDA allocation
dist.init_process_group(backend="nccl")      # env:// reads MASTER_ADDR, RANK, WORLD_SIZE
rank = dist.get_rank()                       # 0-15 across two nodes

model = build_conformer().cuda()             # lands on GPU local_rank
model = DDP(model, device_ids=[local_rank])

train(model)

if rank == 0:
    torch.save(model.module.state_dict(), "asr-conformer.pt")
dist.barrier()                               # nobody reads the file before it exists
dist.destroy_process_group()

go deeper

for a junior

Recall what torchrun exports — RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT — and that LOCAL_RANK, not RANK, picks the GPU.

for a middle

Explain env:// initialisation, why a CUDA process defaults to device 0, and how the old launch module's --local-rank argument differs from torchrun's environment variables.

for a senior

Diagnose the every-rank-on-GPU-0 failure from its symptoms, and show a static multi-node launch with node ranks, the master address and rank-0-only saving behind a barrier.

for a principal

Argue when a fixed torchrun launch is enough and when a team should standardise on init_device_mesh so later sharding or tensor parallelism reuses the same setup.

## What torchrun starts `torchrun` (the console script for `python -m torch.distributed.run`) is the launcher PyTorch ships for multi-process training. On each node it starts `--nproc-per-node` **worker processes** — for GPU training, one per GPU — and gives every worker the information it needs to join one **process group**: the set of processes that run collectives together. `DistributedDataParallel` (DDP) then keeps a full model replica in each worker and all-reduces gradients across the group. The script itself is ordinary single-GPU code plus a few lines of setup. Everything that differs between workers arrives through **environment variables**, not command-line arguments. ## The variables the script reads | Variable | Meaning | Two nodes of eight GPUs | |---|---|---| | `RANK` | global rank of this worker | 0-15 | | `LOCAL_RANK` | rank within this node | 0-7 | | `WORLD_SIZE` | total workers in the job | 16 | | `LOCAL_WORLD_SIZE` | workers on this node, equal to `--nproc-per-node` | 8 | | `MASTER_ADDR` / `MASTER_PORT` | host and port of the TCP store that rank 0 hosts | `node-0.internal.example`, 29500 by default | - `dist.init_process_group(backend="nccl")` with no `init_method` and no `store` assumes **`env://`**: it reads `MASTER_ADDR`, `MASTER_PORT`, `RANK` and `WORLD_SIZE` itself, so the script never parses them. - **`RANK`** decides who does once-per-job work: logging, writing a checkpoint, uploading metrics. - **`LOCAL_RANK`** decides which GPU this process drives. It is the only one of these that maps to a device index. - `RANK` is not a stable identity across launcher restarts; derive nothing permanent from it. ## Why `set_device(LOCAL_RANK)` comes first A CUDA process uses **device 0** until told otherwise, and in its default mode torchrun passes `CUDA_VISIBLE_DEVICES` through unchanged, so on an unrestricted node every worker sees all eight GPUs. A script that calls `model.cuda()` without pinning a device therefore puts all eight replicas on GPU 0. The symptoms are an out-of-memory error on that card, seven idle GPUs in `nvidia-smi`, and — once DDP's constructor runs its first collective — NCCL refusing to build a communicator in which several ranks use the same GPU. The fix is one line, before any CUDA allocation: 1. `local_rank = int(os.environ["LOCAL_RANK"])` 2. `torch.cuda.set_device(local_rank)` 3. `dist.init_process_group(backend="nccl", device_id=local_rank)` — `device_id` is optional; under NCCL it builds the communicator immediately, so a bad setup fails at init rather than at the first collective. 4. `model = DDP(model.cuda(), device_ids=[local_rank])` PyTorch 2.14's torchrun also has a `--virtual-local-rank` mode in which each worker sees only its own GPU (as device 0, with `LOCAL_RANK` set to 0). It is opt-in; the default behaviour is the one above. ## A static two-node launch For a fixed-size job with no restarts, run the same command on both nodes, changing only `--node-rank`: ```bash # on node-0 (use --node-rank=1 on the second node) torchrun --nnodes=2 --nproc-per-node=8 \ --node-rank=0 --master-addr=node-0.internal.example --master-port=29500 \ train_asr.py --config conformer.yaml ``` `--master-addr` and `--master-port` are used only by the static rendezvous; `--standalone` replaces them on a single node by starting a local store on a free port. ## Old launcher, new launcher In PyTorch 2.14, `python -m torch.distributed.launch` still runs but raises a deprecation warning saying it will be removed and to use torchrun. The practical difference is how the local rank arrives: - The old module passed `--local-rank=<n>` to the script as an argument unless `--use-env` was given. - `torchrun` behaves as if `--use-env` were always set: it exports `LOCAL_RANK` and appends no argument, so a script that requires `--local_rank` on its command line must switch to reading the environment. ## Checking the setup in a minute Before a long run, confirm the wiring rather than trusting it: - Log `RANK`, `LOCAL_RANK`, `WORLD_SIZE` and `torch.cuda.current_device()` from every worker at start-up; on each node the local ranks should be 0-7 and each should report a different device. - `nvidia-smi` on each node should show exactly one training process per GPU, with similar memory use. - torchrun's `--local-ranks-filter` restricts which local ranks reach the console, so sixteen copies of every log line do not bury the one that matters. - A first step that runs noticeably slower than a single-GPU step, or memory growing on one card only, usually means a device was not pinned. ## Common mistakes - Passing `RANK` to `set_device`: on node 1 that asks for GPUs 8-15, which do not exist. - Calling `set_device` after the model or optimizer already allocated on GPU 0. - Hard-coding `WORLD_SIZE` or the master address in the script instead of letting `env://` read them. - Saving from every rank, so sixteen processes race to write one file.

  • Where does init_device_mesh fit if the script already calls init_process_group?
    `init_device_mesh("cuda", (16,))` from `torch.distributed.device_mesh` builds a `DeviceMesh` over the ranks; it creates the default process group if none exists and, if no device was set, pins one from `LOCAL_RANK`. DDP accepts a one-dimensional mesh through `device_mesh=`, while FSDP2 and tensor parallelism take meshes, so code that may later grow a second dimension starts from it.
  • How do you save a checkpoint from a DDP job so that one file is written and the others can load it?
    Write `model.module.state_dict()` from rank 0 only — the inner module, so the keys carry no `module.` prefix — then call `dist.barrier()` so no rank reads a half-written file. Ranks that load it pass a `map_location` that maps GPU 0 to their own device; otherwise every tensor lands back on GPU 0.
  • A script written for python -m torch.distributed.launch parses --local_rank; what changes under torchrun?
    torchrun exports `LOCAL_RANK` and appends no `--local-rank` argument, as if the old `--use-env` flag were always on. An argparse that requires `--local_rank` fails or reads nothing, so the script switches to `int(os.environ["LOCAL_RANK"])`. PyTorch 2.14's deprecation message for the old module says exactly this.

  • torchrun splits each batch across the GPUs from a single process
  • RANK is the GPU index to pass to torch.cuda.set_device
  • torchrun hides the other GPUs from each worker by default
  • init_process_group needs the master address passed as an argument under torchrun
  • python -m torch.distributed.launch is the current launcher

Assumes PyTorch 2.14: torchrun's hyphenated flags, the --virtual-local-rank option and the deprecation of torch.distributed.launch are as in the v2.14.1 source.

open as a page

With PyTorch's DistributedSampler in a DDP training loop, what goes wrong when sampler.set_epoch(epoch) is never called, and why?

level: middleimportance: must knowfreq 24%

basics

~20 s

DistributedSampler shuffles with a generator seeded by seed plus epoch, and its epoch stays 0 until set_epoch is called. Without the call every epoch replays one permutation, so each rank sees the same shard in the same order.

open as a page

Your 16-GPU DDP validation pass using PyTorch's DistributedSampler reports a different word error rate than the single-GPU run on the same checkpoint — why?

level: middleimportance: should knowfreq 12%

basics

~20 s

DistributedSampler pads the validation set with repeated utterances so every rank gets the same count, so some are scored twice; drop_last=True drops some instead. Averaging per-rank error rates is a separate, second error. Shard without padding and all-reduce raw counts.

open as a page

A PyTorch DDP model whose auxiliary CTC head runs only on some steps raises a prior-iteration reduction error; what fixes it, at what cost?

level: middleimportance: should knowfreq 11%

basics

~20 s

On steps that skip the head, its parameters get no gradient, so their bucket never finishes reducing. Set find_unused_parameters=True and pay a graph traversal every iteration, or run the head every step with its loss weighted to zero.

open as a page

In PyTorch DDP with gradient accumulation over four micro-batches, how should no_sync be used, and what goes wrong when only backward sits inside it?

level: middleimportance: should knowfreq 16%

basics

~20 s

Run the forward and backward of micro-batches one to three inside model.no_sync(), so gradients only accumulate locally, and the fourth outside it, so its backward all-reduces the summed gradients once. A forward run outside the context still syncs.

open as a page

In PyTorch 2.14 DDP, how do gradient buckets overlap the all-reduce with backward, and when would you change bucket_cap_mb or gradient_as_bucket_view?

level: seniorimportance: should knowfreq 9%

basics

~20 s

DDP packs gradients into buckets, 25 MiB by default with a smaller first one, and launches each bucket's all-reduce as soon as its gradients are ready, while backward continues. Tune bucket_cap_mb by measuring overlap; gradient_as_bucket_view saves one gradient copy.

open as a page