skip to content

Collectives & Interconnects

NCCL runs all-reduce, all-gather and reduce-scatter as ring or tree algorithms over NVLink, InfiniBand or RoCE. Interviewers ask you to estimate all-reduce cost and to debug an NCCL hang.

on this pageshow

questions

6

In PyTorch's torch.distributed, when do you use the NCCL backend, when Gloo, and why do GPU jobs often keep both?

level: juniorimportance: must knowfreq 30%

answer

  1. which tensors each one can touch
  2. CUDA tensors against CPU tensors
  3. InfiniBand and GPUDirect only through one
  4. a host-side barrier needs a CPU backend
  5. one group, two device-qualified backends

basics

~10 s

NCCL is the backend for collectives on CUDA tensors, over NVLink, InfiniBand or RoCE; Gloo runs collectives on CPU tensors over TCP. GPU jobs keep a Gloo group for CPU-side work such as monitored_barrier.

solid answer

~40 s

Use `nccl` for anything on CUDA tensors: PyTorch's guidance names it the only built-in backend that supports InfiniBand and GPUDirect, and it moves data over NVLink inside a node. In PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so a collective on a CPU tensor needs another backend. `gloo` handles CPU tensors over TCP sockets and also runs most collectives on CUDA tensors, only more slowly, which makes it the fallback when NCCL misbehaves on Ethernet. GPU jobs keep Gloo for host-side work: `monitored_barrier` requires a CPU-capable backend, and counters kept on the CPU need one too. One group can carry both with `backend="cpu:gloo,cuda:nccl"`, and calling `init_process_group()` with neither a backend nor a `device_id` creates both.

go deeper

for a junior

Recall the one-line rule: NCCL for CUDA tensors, Gloo for CPU tensors, and be able to pass the right backend string to init_process_group for a GPU job.

for a middle

Explain why a GPU job still creates a Gloo group, with monitored_barrier and CPU-side tensors as examples, and show the cpu:gloo,cuda:nccl form that routes each tensor by device.

for a senior

Show you can use Gloo as a diagnostic, a host-side barrier that names the missing rank or a fallback run on Ethernet when NCCL is suspect, without leaving it on the training path.

for a principal

Weigh the operational cost of two transports per job: two interface variables, two timeouts, two failure modes. Decide which CPU-side traffic deserves a Gloo group and which belongs on the GPU.

## What a backend is in torch.distributed `torch.distributed` defines the **collectives** a training job calls: `all_reduce`, `all_gather`, `reduce_scatter`, `broadcast`, `all_to_all`, `barrier` and the point-to-point `send` / `recv`. A **backend** is the library that actually moves the bytes for those calls. On Linux, PyTorch 2.14 builds two backends by default: **NCCL** (only in CUDA builds) and **Gloo**. **MPI** is available only when PyTorch is built from source on a host with MPI installed, and `xccl` plays NCCL's role for Intel XPU GPUs. You choose with the `backend` argument of `init_process_group`, or per subgroup with `new_group(backend=...)`. ## What each backend can run | | NCCL | Gloo | |---|---|---| | Tensor devices | CUDA only | CPU, and CUDA for most collectives | | Transports | NVLink or PCIe peer-to-peer, shared memory, InfiniBand or RoCE verbs, IP sockets | TCP sockets | | InfiniBand and GPUDirect | Supported | Not supported | | `all_to_all` | On CUDA tensors | Not supported | | `send` / `recv` / `barrier` | On CUDA tensors | On CPU tensors only | | Interface variable | `NCCL_SOCKET_IFNAME` | `GLOO_SOCKET_IFNAME` | | Default timeout in PyTorch 2.14 | 10 minutes | 30 minutes | The capability map in PyTorch 2.14's `distributed_c10d.py` says it in one line: Gloo is registered for `cpu` and `cuda`, NCCL for `cuda` alone. ## The rule of thumb and why it holds - **CUDA GPU training: NCCL.** NCCL implements each collective as a single CUDA kernel that both communicates and reduces, drives NVLink inside a node, and uses InfiniBand or RoCE verbs with **GPUDirect RDMA** between nodes, so gradients need not pass through host memory. - **GPU hosts on Ethernet: still NCCL.** PyTorch's documentation calls Gloo the fallback when NCCL gives trouble there, and notes that Gloo is slower than NCCL for GPUs. - **CPU training: Gloo**, or MPI on CPU hosts with InfiniBand that lack IP over InfiniBand. ## Why GPU jobs still create a Gloo group 1. **`monitored_barrier`.** It is a host-side barrier built from `send` / `recv` acknowledgements that lets rank 0 report which rank failed to arrive within the timeout. PyTorch rejects a group without a CPU-capable backend for it. 2. **CPU tensors.** A step counter, a flag or a metric kept on the CPU has no backend in an NCCL-only group; you either move it to the GPU or give the group a Gloo side. 3. **Debugging.** Rerunning a small reproduction on Gloo tells you whether a failure is specific to NCCL's transports. ## Setting it up in PyTorch 2.14 - `init_process_group(backend="nccl")` creates an NCCL-only group. - `init_process_group(backend="cpu:gloo,cuda:nccl")` creates one group whose collectives dispatch by the tensor's device. - `init_process_group()` with neither a backend nor a `device_id` creates both Gloo and NCCL; the docs mark multi-backend support as experimental. - `init_process_group(device_id=torch.device("cuda", local_rank))` with no backend takes the backend from the device type (NCCL for CUDA) and creates the NCCL communicator immediately instead of at the first collective. - `new_group(backend="gloo")` adds a separate CPU group beside an NCCL default group. A device-qualified group in practice, after each process has selected its GPU: ```python import torch import torch.distributed as dist dist.init_process_group(backend="cpu:gloo,cuda:nccl") grad = torch.ones(1024, device="cuda") dist.all_reduce(grad) # CUDA tensor: runs on NCCL steps = torch.tensor([128]) dist.all_reduce(steps) # CPU tensor: runs on Gloo dist.monitored_barrier() # host-side barrier: uses the Gloo side ``` Each backend also picks its own network interface: `NCCL_SOCKET_IFNAME` steers NCCL, `GLOO_SOCKET_IFNAME` steers Gloo, and pinning one leaves the other on automatic selection. Each also has its own default timeout, so a stuck Gloo call waits three times as long as a stuck NCCL one unless you pass `timeout=` explicitly. ## What interviewers listen for - That the choice follows the **tensor's device**, not the model or the framework on top. - That Gloo **can** run many collectives on CUDA tensors, slowly, rather than "Gloo is CPU-only". - That NCCL **cannot** take CPU tensors at all. - A concrete reason to keep Gloo in a GPU job, with `monitored_barrier` as the usual example. - The device-qualified backend string, which avoids managing two groups by hand.

  • What happens if you call all_reduce on a CPU tensor in a group created with backend="nccl"?
    It fails: in PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so the group has no backend for a CPU tensor. Move the tensor to the GPU, which is what you want for anything large, or create the group as `cpu:gloo,cuda:nccl`, or add a separate `new_group(backend="gloo")` for CPU-side collectives.
  • Which variable steers each backend's network interface on a node with several interfaces?
    NCCL reads `NCCL_SOCKET_IFNAME` and Gloo reads `GLOO_SOCKET_IFNAME`, so pinning one leaves the other on automatic selection. Gloo accepts a comma-separated list and round-robins operations across those interfaces, and every process must list the same number of them.

  • Gloo cannot run any collective on a GPU tensor.
  • NCCL can reduce CPU tensors too, just more slowly.
  • Gloo is the faster choice for multi-node GPU training on InfiniBand.
  • Once NCCL is chosen, a GPU job never needs a Gloo group.
  • The backend choice also decides how DistributedSampler splits the data.

Assumes PyTorch 2.14: its capability map registers NCCL for CUDA only and Gloo for CPU and CUDA, and an omitted backend creates both.

open as a page

Across 32 GPUs, how much data does each rank send in NCCL's all-reduce, all-gather, reduce-scatter and all-to-all of a 400 MB buffer?

level: middleimportance: must knowfreq 22%

basics

~10 s

With S the full 400 MB array and n = 32, all-reduce sends 2(n-1)/n x S, about 775 MB per rank; reduce-scatter, all-gather and all-to-all each send (n-1)/n x S, about 387.5 MB.

open as a page

On a new multi-node cluster the first NCCL collective of a PyTorch job never returns; how do NCCL_DEBUG and NCCL_SOCKET_IFNAME help you find why?

level: middleimportance: should knowfreq 14%

basics

~10 s

Set NCCL_DEBUG=INFO on every rank and read which interface each one bootstraps on and which network it chose; if NCCL picked an interface the other nodes cannot reach, pin the routable one with NCCL_SOCKET_IFNAME.

open as a page

An nccl-tests all_reduce_perf run on four RoCE-connected 8-GPU nodes shows a fraction of the fabric's rating; how do you read busbw and find the cause?

level: seniorimportance: should knowfreq 8%

basics

~10 s

Compare busbw, not algbw, with the per-GPU link rating; for all-reduce busbw = algbw x 2(n-1)/n. If large-message busbw is still low, NCCL_DEBUG=INFO shows a socket fallback, too few adapters or missing GPUDirect RDMA.

open as a page

How does NCCL choose between ring and tree for an all-reduce, and when, if ever, should you set NCCL_ALGO or NCCL_PROTO?

level: seniorimportance: should knowfreq 9%

basics

~20 s

NCCL models each algorithm and protocol's latency and bandwidth for the message size and detected topology, and picks the fastest: tree often wins small all-reduces across many nodes, ring usually large ones. Override NCCL_ALGO or NCCL_PROTO only to experiment or debug.

open as a page

With PyTorch 2.14's NCCL process group, a job dies on a watchdog timeout after ten minutes; how do you find the rank that diverged?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

Read the flight recorder: PyTorch 2.14 records each rank's recent NCCL collectives and dumps them on a watchdog timeout. Comparing the last sequence number and operation across ranks shows the rank that issued a different or missing collective.

open as a page