skip to content

Distributed Training Infrastructure

DDP, FSDP and DeepSpeed spread one training job over many GPUs, NCCL carries the gradients, and schedulers keep the job placed and alive. GPU-owning teams probe it from layout to hang.

on this pageshow

explore

questions

page 1 of 2

In PyTorch's torch.distributed, when do you use the NCCL backend, when Gloo, and why do GPU jobs often keep both?

level: juniorimportance: must knowfreq 30%

answer

  1. which tensors each one can touch
  2. CUDA tensors against CPU tensors
  3. InfiniBand and GPUDirect only through one
  4. a host-side barrier needs a CPU backend
  5. one group, two device-qualified backends

basics

~10 s

NCCL is the backend for collectives on CUDA tensors, over NVLink, InfiniBand or RoCE; Gloo runs collectives on CPU tensors over TCP. GPU jobs keep a Gloo group for CPU-side work such as monitored_barrier.

solid answer

~40 s

Use `nccl` for anything on CUDA tensors: PyTorch's guidance names it the only built-in backend that supports InfiniBand and GPUDirect, and it moves data over NVLink inside a node. In PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so a collective on a CPU tensor needs another backend. `gloo` handles CPU tensors over TCP sockets and also runs most collectives on CUDA tensors, only more slowly, which makes it the fallback when NCCL misbehaves on Ethernet. GPU jobs keep Gloo for host-side work: `monitored_barrier` requires a CPU-capable backend, and counters kept on the CPU need one too. One group can carry both with `backend="cpu:gloo,cuda:nccl"`, and calling `init_process_group()` with neither a backend nor a `device_id` creates both.

go deeper

for a junior

Recall the one-line rule: NCCL for CUDA tensors, Gloo for CPU tensors, and be able to pass the right backend string to init_process_group for a GPU job.

for a middle

Explain why a GPU job still creates a Gloo group, with monitored_barrier and CPU-side tensors as examples, and show the cpu:gloo,cuda:nccl form that routes each tensor by device.

for a senior

Show you can use Gloo as a diagnostic, a host-side barrier that names the missing rank or a fallback run on Ethernet when NCCL is suspect, without leaving it on the training path.

for a principal

Weigh the operational cost of two transports per job: two interface variables, two timeouts, two failure modes. Decide which CPU-side traffic deserves a Gloo group and which belongs on the GPU.

## What a backend is in torch.distributed `torch.distributed` defines the **collectives** a training job calls: `all_reduce`, `all_gather`, `reduce_scatter`, `broadcast`, `all_to_all`, `barrier` and the point-to-point `send` / `recv`. A **backend** is the library that actually moves the bytes for those calls. On Linux, PyTorch 2.14 builds two backends by default: **NCCL** (only in CUDA builds) and **Gloo**. **MPI** is available only when PyTorch is built from source on a host with MPI installed, and `xccl` plays NCCL's role for Intel XPU GPUs. You choose with the `backend` argument of `init_process_group`, or per subgroup with `new_group(backend=...)`. ## What each backend can run | | NCCL | Gloo | |---|---|---| | Tensor devices | CUDA only | CPU, and CUDA for most collectives | | Transports | NVLink or PCIe peer-to-peer, shared memory, InfiniBand or RoCE verbs, IP sockets | TCP sockets | | InfiniBand and GPUDirect | Supported | Not supported | | `all_to_all` | On CUDA tensors | Not supported | | `send` / `recv` / `barrier` | On CUDA tensors | On CPU tensors only | | Interface variable | `NCCL_SOCKET_IFNAME` | `GLOO_SOCKET_IFNAME` | | Default timeout in PyTorch 2.14 | 10 minutes | 30 minutes | The capability map in PyTorch 2.14's `distributed_c10d.py` says it in one line: Gloo is registered for `cpu` and `cuda`, NCCL for `cuda` alone. ## The rule of thumb and why it holds - **CUDA GPU training: NCCL.** NCCL implements each collective as a single CUDA kernel that both communicates and reduces, drives NVLink inside a node, and uses InfiniBand or RoCE verbs with **GPUDirect RDMA** between nodes, so gradients need not pass through host memory. - **GPU hosts on Ethernet: still NCCL.** PyTorch's documentation calls Gloo the fallback when NCCL gives trouble there, and notes that Gloo is slower than NCCL for GPUs. - **CPU training: Gloo**, or MPI on CPU hosts with InfiniBand that lack IP over InfiniBand. ## Why GPU jobs still create a Gloo group 1. **`monitored_barrier`.** It is a host-side barrier built from `send` / `recv` acknowledgements that lets rank 0 report which rank failed to arrive within the timeout. PyTorch rejects a group without a CPU-capable backend for it. 2. **CPU tensors.** A step counter, a flag or a metric kept on the CPU has no backend in an NCCL-only group; you either move it to the GPU or give the group a Gloo side. 3. **Debugging.** Rerunning a small reproduction on Gloo tells you whether a failure is specific to NCCL's transports. ## Setting it up in PyTorch 2.14 - `init_process_group(backend="nccl")` creates an NCCL-only group. - `init_process_group(backend="cpu:gloo,cuda:nccl")` creates one group whose collectives dispatch by the tensor's device. - `init_process_group()` with neither a backend nor a `device_id` creates both Gloo and NCCL; the docs mark multi-backend support as experimental. - `init_process_group(device_id=torch.device("cuda", local_rank))` with no backend takes the backend from the device type (NCCL for CUDA) and creates the NCCL communicator immediately instead of at the first collective. - `new_group(backend="gloo")` adds a separate CPU group beside an NCCL default group. A device-qualified group in practice, after each process has selected its GPU: ```python import torch import torch.distributed as dist dist.init_process_group(backend="cpu:gloo,cuda:nccl") grad = torch.ones(1024, device="cuda") dist.all_reduce(grad) # CUDA tensor: runs on NCCL steps = torch.tensor([128]) dist.all_reduce(steps) # CPU tensor: runs on Gloo dist.monitored_barrier() # host-side barrier: uses the Gloo side ``` Each backend also picks its own network interface: `NCCL_SOCKET_IFNAME` steers NCCL, `GLOO_SOCKET_IFNAME` steers Gloo, and pinning one leaves the other on automatic selection. Each also has its own default timeout, so a stuck Gloo call waits three times as long as a stuck NCCL one unless you pass `timeout=` explicitly. ## What interviewers listen for - That the choice follows the **tensor's device**, not the model or the framework on top. - That Gloo **can** run many collectives on CUDA tensors, slowly, rather than "Gloo is CPU-only". - That NCCL **cannot** take CPU tensors at all. - A concrete reason to keep Gloo in a GPU job, with `monitored_barrier` as the usual example. - The device-qualified backend string, which avoids managing two groups by hand.

  • What happens if you call all_reduce on a CPU tensor in a group created with backend="nccl"?
    It fails: in PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so the group has no backend for a CPU tensor. Move the tensor to the GPU, which is what you want for anything large, or create the group as `cpu:gloo,cuda:nccl`, or add a separate `new_group(backend="gloo")` for CPU-side collectives.
  • Which variable steers each backend's network interface on a node with several interfaces?
    NCCL reads `NCCL_SOCKET_IFNAME` and Gloo reads `GLOO_SOCKET_IFNAME`, so pinning one leaves the other on automatic selection. Gloo accepts a comma-separated list and round-robins operations across those interfaces, and every process must list the same number of them.

  • Gloo cannot run any collective on a GPU tensor.
  • NCCL can reduce CPU tensors too, just more slowly.
  • Gloo is the faster choice for multi-node GPU training on InfiniBand.
  • Once NCCL is chosen, a GPU job never needs a Gloo group.
  • The backend choice also decides how DistributedSampler splits the data.

Assumes PyTorch 2.14: its capability map registers NCCL for CUDA only and Gloo for CPU and CUDA, and an omitted backend creates both.

open as a page

What does model FLOPs utilization (MFU) measure for a transformer training run, and how do you compute it?

level: juniorimportance: must knowfreq 30%

basics

~20 s

MFU is the share of the GPUs' dense peak FLOP rate spent on the model's own arithmetic: tokens per second times about 6 x parameters (plus an attention term), divided by GPU count times per-GPU peak.

open as a page

When torchrun launches a PyTorch DDP job, what does the training script read from the environment, and why must it call torch.cuda.set_device(LOCAL_RANK)?

level: juniorimportance: must knowfreq 32%

basics

~20 s

torchrun starts one process per GPU and exports RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT, which init_process_group reads through env://. Each process must pin itself to GPU LOCAL_RANK, or every rank on a node lands on GPU 0.

open as a page

Across 32 GPUs, how much data does each rank send in NCCL's all-reduce, all-gather, reduce-scatter and all-to-all of a 400 MB buffer?

level: middleimportance: must knowfreq 22%

basics

~10 s

With S the full 400 MB array and n = 32, all-reduce sends 2(n-1)/n x S, about 775 MB per rank; reduce-scatter, all-gather and all-to-all each send (n-1)/n x S, about 387.5 MB.

open as a page

In DeepSpeed ZeRO, what do stages 1, 2 and 3 each partition, and which stage fits a 70B model on 256 80 GB GPUs?

level: middleimportance: must knowfreq 30%

basics

~20 s

Stage 1 partitions optimizer states across data-parallel ranks, stage 2 adds the gradients and stage 3 adds the 16-bit weights. For 70B parameters on 256 GPUs only stage 3 fits: about 4.4 GB of model state per GPU.

open as a page

For notebook users on 80 GB GPUs managed by the NVIDIA GPU Operator, when do you pick MIG slices over time-sliced replicas, and what does each isolate?

level: middleimportance: must knowfreq 18%

basics

~20 s

MIG splits a GPU into hardware instances with their own memory and fault isolation, in fixed profiles like 1g.10gb. Time-slicing hands out replicas of one GPU that share its memory and compute, isolating nothing. Pick MIG when tenants must not disturb each other.

open as a page

With torch.profiler in PyTorch 2.14, how do you capture a few representative training steps, and what do you look for in the trace?

level: middleimportance: must knowfreq 18%

basics

~10 s

Wrap the loop in torch.profiler.profile with schedule(skip_first, wait, warmup, active, repeat) and call prof.step() each iteration; then read the exported trace for gaps on the GPU stream and what the CPU did meanwhile.

open as a page

With PyTorch's torchrun and --max-restarts=3, what is actually restarted when one worker on one node crashes, and what must the training script do?

level: middleimportance: must knowfreq 18%

basics

~20 s

torchrun stops every worker in the job and reruns the script on all nodes after a fresh rendezvous, up to --max-restarts times (default 0). The script must load its latest checkpoint at startup, or it starts again from step zero.

open as a page

With PyTorch's DistributedSampler in a DDP training loop, what goes wrong when sampler.set_epoch(epoch) is never called, and why?

level: middleimportance: must knowfreq 24%

basics

~20 s

DistributedSampler shuffles with a generator seeded by seed plus epoch, and its epoch stays 0 until set_epoch is called. Without the call every epoch replays one permutation, so each rank sees the same shard in the same order.

open as a page

A 16-pod pretraining job on a shared Kubernetes GPU cluster has 11 pods running and holding GPUs while 5 stay Pending; how do Kueue and Volcano each prevent this?

level: seniorimportance: must knowfreq 13%

basics

~20 s

Kueue admits a Workload only when quota for every pod is free, can check node placement with topology-aware scheduling, and with waitForPodsReady evicts and requeues a Workload not fully ready in time. Volcano binds none of a PodGroup's pods until minMember can be placed.

open as a page

What does the NVIDIA GPU Operator install and manage on a Kubernetes cluster's GPU nodes, and which resource holds its configuration?

level: juniorimportance: should knowfreq 15%

basics

~20 s

The NVIDIA GPU Operator runs the whole GPU software stack as containers on each GPU node: driver, container toolkit, device plugin, GPU Feature Discovery, DCGM exporter, MIG Manager and a validator. One ClusterPolicy resource declares that desired state.

open as a page

On a new multi-node cluster the first NCCL collective of a PyTorch job never returns; how do NCCL_DEBUG and NCCL_SOCKET_IFNAME help you find why?

level: middleimportance: should knowfreq 14%

basics

~10 s

Set NCCL_DEBUG=INFO on every rank and read which interface each one bootstraps on and which network it chose; if NCCL picked an interface the other nodes cannot reach, pin the routable one with NCCL_SOCKET_IFNAME.

open as a page

What does PyTorch 2.14's FSDP2 fully_shard do to a model's parameters, and what changes for code written against FSDP1?

level: middleimportance: should knowfreq 15%

basics

~20 s

fully_shard swaps each parameter in place for a DTensor sharded on dim 0 across the mesh and all-gathers each call's group around forward and backward. FSDP1's wrapper, FlatParameter and ShardingStrategy give way to per-block calls and reshard_after_forward.

open as a page

In Kueue 0.20, how do a ResourceFlavor, a ClusterQueue and a LocalQueue decide when a team's GPU job is admitted, and how does a cohort let it borrow?

level: middleimportance: should knowfreq 12%

basics

~20 s

Kueue admits whole jobs against quota: a ResourceFlavor names a class of nodes, a ClusterQueue holds nominalQuota per flavor and resource, and a LocalQueue routes a namespace's jobs to it. ClusterQueues sharing a cohortName can borrow each other's unused quota.

open as a page

With the NVIDIA GPU Operator, how do the MIG single and mixed strategies change the resource a pod requests, and how do you repartition a node?

level: middleimportance: should knowfreq 8%

basics

~20 s

Under the single strategy every GPU on a node has the same MIG profile and each instance is advertised as nvidia.com/gpu; under mixed, each profile becomes its own resource such as nvidia.com/mig-3g.40gb. Repartition by changing the node's nvidia.com/mig.config label.

open as a page

In PyTorch 2.14, what does torch.utils.checkpoint do when use_reentrant is omitted, and why pass use_reentrant=False?

level: middleimportance: should knowfreq 12%

basics

~10 s

In PyTorch 2.14, omitting use_reentrant warns and falls back to the reentrant variant. Pass use_reentrant=False: it handles frozen inputs, keyword arguments, nested inputs and torch.autograd.grad, stops recomputing early, and enables selective checkpointing.

open as a page

On a Slurm cluster, when do you launch a 64-node PyTorch job as one torchrun per node versus one srun task per GPU?

level: middleimportance: should knowfreq 14%

basics

~20 s

Run one torchrun per node (srun --ntasks-per-node=1) when you want torchrun's c10d rendezvous and in-place restarts; run one srun task per GPU (--ntasks-per-node=8) when Slurm should own every process and you map SLURM_PROCID and SLURM_LOCALID to ranks yourself.

open as a page

Your 16-GPU DDP validation pass using PyTorch's DistributedSampler reports a different word error rate than the single-GPU run on the same checkpoint — why?

level: middleimportance: should knowfreq 12%

basics

~20 s

DistributedSampler pads the validation set with repeated utterances so every rank gets the same count, so some are scored twice; drop_last=True drops some instead. Averaging per-rank error rates is a separate, second error. Shard without padding and all-reduce raw counts.

open as a page

A PyTorch DDP model whose auxiliary CTC head runs only on some steps raises a prior-iteration reduction error; what fixes it, at what cost?

level: middleimportance: should knowfreq 11%

basics

~20 s

On steps that skip the head, its parameters get no gradient, so their bucket never finishes reducing. Set find_unused_parameters=True and pay a graph traversal every iteration, or run the head every step with its loss weighted to zero.

open as a page

In PyTorch DDP with gradient accumulation over four micro-batches, how should no_sync be used, and what goes wrong when only backward sits inside it?

level: middleimportance: should knowfreq 16%

basics

~20 s

Run the forward and backward of micro-batches one to three inside model.no_sync(), so gradients only accumulate locally, and the fourth outside it, so its backward all-reduces the summed gradients once. A forward run outside the context still syncs.

open as a page

An nccl-tests all_reduce_perf run on four RoCE-connected 8-GPU nodes shows a fraction of the fabric's rating; how do you read busbw and find the cause?

level: seniorimportance: should knowfreq 8%

basics

~10 s

Compare busbw, not algbw, with the per-GPU link rating; for all-reduce busbw = algbw x 2(n-1)/n. If large-message busbw is still low, NCCL_DEBUG=INFO shows a socket fallback, too few adapters or missing GPUDirect RDMA.

open as a page

How does NCCL choose between ring and tree for an all-reduce, and when, if ever, should you set NCCL_ALGO or NCCL_PROTO?

level: seniorimportance: should knowfreq 9%

basics

~20 s

NCCL models each algorithm and protocol's latency and bandwidth for the message size and detected topology, and picks the fastest: tree often wins small all-reduces across many nodes, ring usually large ones. Override NCCL_ALGO or NCCL_PROTO only to experiment or debug.

open as a page

With Megatron-Core 0.19, how would you set TP, PP, CP and DP for a 70B dense decoder on 32 nodes of eight 80 GB GPUs?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Tensor parallel 8 with sequence parallelism inside each node, pipeline parallel 4 across nodes with virtual stages, context parallel 1 at 8K tokens, leaving data parallel 256 / (8 x 4 x 1) = 8, plus the distributed optimizer.

open as a page

With expert parallelism in Megatron-Core or DeepSpeed, what all-to-all traffic does each MoE layer of a 64-expert model add, and where should the expert-parallel group sit?

level: seniorimportance: should knowfreq 6%

basics

~20 s

Each MoE layer adds a dispatch all-to-all sending tokens to the ranks holding their experts and a combine all-to-all bringing results back, mirrored in backward, every micro-batch. Keep the expert-parallel group inside one NVLink node.

open as a page

A torch.compile'd 7B fine-tune in PyTorch 2.14 shows little speedup and slow early steps; how do you find graph breaks and recompiles, and decide if compiling pays?

level: seniorimportance: should knowfreq 10%

basics

~10 s

Run torch._dynamo.explain on the step or compile with fullgraph=True to find graph breaks (prints, .item(), data-dependent branches); stop shape-driven recompiles with bucketing or mark_dynamic; then weigh one-off compile time against per-step savings.

open as a page

How do you capture a CUDA memory snapshot of a PyTorch 2.14 fine-tune that OOMs hours in, and tell fragmentation from a leak in it?

level: seniorimportance: should knowfreq 12%

basics

~10 s

Call torch.cuda.memory._record_memory_history(max_entries=...) at launch, catch the OOM and call _dump_snapshot, then open it in the memory viewer: a rising per-step floor is a leak; segments full of unusable free slivers are fragmentation.

open as a page

A 512-GPU PyTorch run saved with torch.distributed.checkpoint must resume on 256 GPUs after half the cluster goes down — how does the load work, and what else must change?

level: seniorimportance: should knowfreq 9%

basics

~20 s

Distributed Checkpoint reshards at load time: build and shard the model for 256 ranks, call dcp.load, and each rank reads only the saved chunks overlapping its new shards. Global batch, data position and per-rank RNG state are yours to adjust.

open as a page

How do you confirm that one straggler rank is slowing a 512-GPU Slurm training run by 30%, and what do you do about it?

level: seniorimportance: should knowfreq 12%

basics

~20 s

Every collective finishes only when the slowest rank arrives, so one slow rank slows all 512. Time each rank's own compute and data wait, gather the numbers with hostnames, then exclude the outlier's node and resume from the last checkpoint.

open as a page

In PyTorch 2.14 DDP, how do gradient buckets overlap the all-reduce with backward, and when would you change bucket_cap_mb or gradient_as_bucket_view?

level: seniorimportance: should knowfreq 9%

basics

~20 s

DDP packs gradients into buckets, 25 MiB by default with a smaller first one, and launches each bucket's all-reduce as soon as its gradients are ready, while backward continues. Tune bucket_cap_mb by measuring overlap; gradient_as_bucket_view saves one gradient copy.

open as a page

How do you choose the checkpoint interval for a month-long 512-GPU pretraining run that loses a node every few days, and what does PyTorch's DCP async_save change?

level: principalimportance: should knowfreq 8%

basics

~20 s

Balance each save's blocking time against the work lost per failure: an interval near sqrt(2 x blocking cost x mean time between failures) minimises the sum. DCP async_save shrinks the blocking cost to a CPU staging copy, allowing far more frequent saves.

open as a page

showing 1–30 of 36