In PyTorch's torch.distributed, when do you use the NCCL backend, when Gloo, and why do GPU jobs often keep both?
answer
- which tensors each one can touch
- CUDA tensors against CPU tensors
- InfiniBand and GPUDirect only through one
- a host-side barrier needs a CPU backend
- one group, two device-qualified backends
basics
~10 sNCCL is the backend for collectives on CUDA tensors, over NVLink, InfiniBand or RoCE; Gloo runs collectives on CPU tensors over TCP. GPU jobs keep a Gloo group for CPU-side work such as monitored_barrier.
solid answer
~40 sUse `nccl` for anything on CUDA tensors: PyTorch's guidance names it the only built-in backend that supports InfiniBand and GPUDirect, and it moves data over NVLink inside a node. In PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so a collective on a CPU tensor needs another backend. `gloo` handles CPU tensors over TCP sockets and also runs most collectives on CUDA tensors, only more slowly, which makes it the fallback when NCCL misbehaves on Ethernet. GPU jobs keep Gloo for host-side work: `monitored_barrier` requires a CPU-capable backend, and counters kept on the CPU need one too. One group can carry both with `backend="cpu:gloo,cuda:nccl"`, and calling `init_process_group()` with neither a backend nor a `device_id` creates both.
go deeper
Recall the one-line rule: NCCL for CUDA tensors, Gloo for CPU tensors, and be able to pass the right backend string to init_process_group for a GPU job.
Explain why a GPU job still creates a Gloo group, with monitored_barrier and CPU-side tensors as examples, and show the cpu:gloo,cuda:nccl form that routes each tensor by device.
Show you can use Gloo as a diagnostic, a host-side barrier that names the missing rank or a fallback run on Ethernet when NCCL is suspect, without leaving it on the training path.
Weigh the operational cost of two transports per job: two interface variables, two timeouts, two failure modes. Decide which CPU-side traffic deserves a Gloo group and which belongs on the GPU.
## What a backend is in torch.distributed `torch.distributed` defines the **collectives** a training job calls: `all_reduce`, `all_gather`, `reduce_scatter`, `broadcast`, `all_to_all`, `barrier` and the point-to-point `send` / `recv`. A **backend** is the library that actually moves the bytes for those calls. On Linux, PyTorch 2.14 builds two backends by default: **NCCL** (only in CUDA builds) and **Gloo**. **MPI** is available only when PyTorch is built from source on a host with MPI installed, and `xccl` plays NCCL's role for Intel XPU GPUs. You choose with the `backend` argument of `init_process_group`, or per subgroup with `new_group(backend=...)`. ## What each backend can run | | NCCL | Gloo | |---|---|---| | Tensor devices | CUDA only | CPU, and CUDA for most collectives | | Transports | NVLink or PCIe peer-to-peer, shared memory, InfiniBand or RoCE verbs, IP sockets | TCP sockets | | InfiniBand and GPUDirect | Supported | Not supported | | `all_to_all` | On CUDA tensors | Not supported | | `send` / `recv` / `barrier` | On CUDA tensors | On CPU tensors only | | Interface variable | `NCCL_SOCKET_IFNAME` | `GLOO_SOCKET_IFNAME` | | Default timeout in PyTorch 2.14 | 10 minutes | 30 minutes | The capability map in PyTorch 2.14's `distributed_c10d.py` says it in one line: Gloo is registered for `cpu` and `cuda`, NCCL for `cuda` alone. ## The rule of thumb and why it holds - **CUDA GPU training: NCCL.** NCCL implements each collective as a single CUDA kernel that both communicates and reduces, drives NVLink inside a node, and uses InfiniBand or RoCE verbs with **GPUDirect RDMA** between nodes, so gradients need not pass through host memory. - **GPU hosts on Ethernet: still NCCL.** PyTorch's documentation calls Gloo the fallback when NCCL gives trouble there, and notes that Gloo is slower than NCCL for GPUs. - **CPU training: Gloo**, or MPI on CPU hosts with InfiniBand that lack IP over InfiniBand. ## Why GPU jobs still create a Gloo group 1. **`monitored_barrier`.** It is a host-side barrier built from `send` / `recv` acknowledgements that lets rank 0 report which rank failed to arrive within the timeout. PyTorch rejects a group without a CPU-capable backend for it. 2. **CPU tensors.** A step counter, a flag or a metric kept on the CPU has no backend in an NCCL-only group; you either move it to the GPU or give the group a Gloo side. 3. **Debugging.** Rerunning a small reproduction on Gloo tells you whether a failure is specific to NCCL's transports. ## Setting it up in PyTorch 2.14 - `init_process_group(backend="nccl")` creates an NCCL-only group. - `init_process_group(backend="cpu:gloo,cuda:nccl")` creates one group whose collectives dispatch by the tensor's device. - `init_process_group()` with neither a backend nor a `device_id` creates both Gloo and NCCL; the docs mark multi-backend support as experimental. - `init_process_group(device_id=torch.device("cuda", local_rank))` with no backend takes the backend from the device type (NCCL for CUDA) and creates the NCCL communicator immediately instead of at the first collective. - `new_group(backend="gloo")` adds a separate CPU group beside an NCCL default group. A device-qualified group in practice, after each process has selected its GPU: ```python import torch import torch.distributed as dist dist.init_process_group(backend="cpu:gloo,cuda:nccl") grad = torch.ones(1024, device="cuda") dist.all_reduce(grad) # CUDA tensor: runs on NCCL steps = torch.tensor([128]) dist.all_reduce(steps) # CPU tensor: runs on Gloo dist.monitored_barrier() # host-side barrier: uses the Gloo side ``` Each backend also picks its own network interface: `NCCL_SOCKET_IFNAME` steers NCCL, `GLOO_SOCKET_IFNAME` steers Gloo, and pinning one leaves the other on automatic selection. Each also has its own default timeout, so a stuck Gloo call waits three times as long as a stuck NCCL one unless you pass `timeout=` explicitly. ## What interviewers listen for - That the choice follows the **tensor's device**, not the model or the framework on top. - That Gloo **can** run many collectives on CUDA tensors, slowly, rather than "Gloo is CPU-only". - That NCCL **cannot** take CPU tensors at all. - A concrete reason to keep Gloo in a GPU job, with `monitored_barrier` as the usual example. - The device-qualified backend string, which avoids managing two groups by hand.
- What happens if you call all_reduce on a CPU tensor in a group created with backend="nccl"?It fails: in PyTorch 2.14 the NCCL backend is registered for CUDA devices only, so the group has no backend for a CPU tensor. Move the tensor to the GPU, which is what you want for anything large, or create the group as `cpu:gloo,cuda:nccl`, or add a separate `new_group(backend="gloo")` for CPU-side collectives.
- Which variable steers each backend's network interface on a node with several interfaces?NCCL reads `NCCL_SOCKET_IFNAME` and Gloo reads `GLOO_SOCKET_IFNAME`, so pinning one leaves the other on automatic selection. Gloo accepts a comma-separated list and round-robins operations across those interfaces, and every process must list the same number of them.
- Gloo cannot run any collective on a GPU tensor.
- NCCL can reduce CPU tensors too, just more slowly.
- Gloo is the faster choice for multi-node GPU training on InfiniBand.
- Once NCCL is chosen, a GPU job never needs a Gloo group.
- The backend choice also decides how DistributedSampler splits the data.
Assumes PyTorch 2.14: its capability map registers NCCL for CUDA only and Gloo for CPU and CUDA, and an omitted backend creates both.