skip to content

On a new multi-node cluster the first NCCL collective of a PyTorch job never returns; how do NCCL_DEBUG and NCCL_SOCKET_IFNAME help you find why?

level: middleimportance: should knowfreq 14%

answer

  1. the first call is also the setup
  2. make every rank say what it chose
  3. the Bootstrap and Using network lines
  4. ib-prefixed interfaces win by default
  5. a leading = for an exact name

basics

~10 s

Set NCCL_DEBUG=INFO on every rank and read which interface each one bootstraps on and which network it chose; if NCCL picked an interface the other nodes cannot reach, pin the routable one with NCCL_SOCKET_IFNAME.

solid answer

~40 s

The first collective is where NCCL builds its communicator (unless a `device_id` made PyTorch create it eagerly), so a hang there is usually a setup problem. Set `NCCL_DEBUG=INFO`, add `NET` to `NCCL_DEBUG_SUBSYS` (default `INIT,BOOTSTRAP,ENV` in NCCL 2.32), and write per-rank logs with `NCCL_DEBUG_FILE` and its `%h.%p` pattern. Compare the `Bootstrap: Using <interface>:<address>` line and the `Using network IB` or `Using network Socket` line across nodes. With `NCCL_SOCKET_IFNAME` unset, NCCL prefers interfaces whose names start with `ib`, then falls back to any other except loopback and docker ones, which can be a management or virtual interface the other nodes cannot reach. Pin the data interface, for example `NCCL_SOCKET_IFNAME="=ens1f0"`. If setup then completes but transfers stall, `NCCL_IB_DISABLE=1` and `NCCL_P2P_DISABLE=1` isolate the network and intra-node paths for a test run only.

code

bash · 10 lines
bash
# On every node, before the ranks start
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,BOOTSTRAP,ENV,NET
export NCCL_DEBUG_FILE=/shared/nccl-logs/nccl.%h.%p

# After a failed run: which interface and network did each rank pick?
grep -hE "Bootstrap: Using|Using network" /shared/nccl-logs/nccl.*

# Once the logs show the wrong interface, pin the data interface exactly
export NCCL_SOCKET_IFNAME="=ens1f0"

go deeper

for a junior

Recall the two variables: NCCL_DEBUG=INFO makes NCCL explain its choices, and NCCL_SOCKET_IFNAME tells it which network interface to use.

for a middle

Explain why the first collective is where NCCL sets up, read the Bootstrap and Using network lines, and describe NCCL's automatic interface order and the = and ^ syntax for pinning one.

for a senior

Show a method: per-rank log files, compare advertised addresses across nodes, isolate transports with the disable toggles, and remove every toggle once the cause is fixed.

for a principal

Push the fix into the platform, for example a cluster-wide /etc/nccl.conf with the right interface names, so no training team rediscovers it on the next node pool.

## Why the first collective is where it hangs NCCL is not a launcher: it relies on the application's processes and on a CPU-side **bootstrap** over TCP to find its peers. PyTorch creates the NCCL **communicator** lazily, at the first collective on a process group, unless `init_process_group` was given a `device_id`, in which case it is created immediately. So "the first all-reduce never returns" usually means one of three things: 1. The **bootstrap** cannot connect, because a rank advertised an address the others cannot reach. 2. The bootstrap succeeds but the **data transport** (RDMA adapters, peer-to-peer inside the node) cannot connect. 3. The ranks really are inside different collectives, a code problem rather than a network one. The first two leave fingerprints in NCCL's own log; the job eventually dies on PyTorch's process-group timeout either way. ## Make every rank report what it chose - `NCCL_DEBUG=INFO` prints NCCL's debug information. Unset, NCCL prints nothing; `WARN` prints only errors when NCCL calls fail, which a silent hang may never produce. NCCL 2.32 adds an `ATTN` level, `WARN` plus notices. - `NCCL_DEBUG_SUBSYS` filters the INFO output. Its default in NCCL 2.32 is `INIT,BOOTSTRAP,ENV`; add `NET` for transport detail and `GRAPH` for topology detection. - `NCCL_DEBUG_FILE` writes the log to a file; `%h` and `%p` expand to host name and process ID, so 32 ranks do not overwrite one file. ## Reading the log | Line in the INFO output | What it tells you | |---|---| | `NCCL_SOCKET_IFNAME set by environment to ...` | The interface filter actually reached this rank | | `Bootstrap: Using <if>:<address>` | The interface and address this rank advertised to its peers | | `Using network IB` or `Using network Socket` | Whether inter-node data goes over RDMA verbs or IP sockets | | `NET/IB : Using [0]mlx5_0:1/RoCE ...; OOB <if>:<address>` | The RDMA adapters found, their link layer, and the socket interface used for connection setup | | `Bootstrap : no socket interface found` | No interface survived the selection or the filter at all | If the logs stop right after the `Bootstrap` line, compare the advertised addresses: one node offering a management, bridge or link-local address is the classic new-cluster hang. ## How NCCL picks an interface, and how to pin one With `NCCL_SOCKET_IFNAME` unset, NCCL 2.32 selects automatically: 1. interfaces whose names start with `ib`; 2. otherwise, if `NCCL_COMM_ID` is set, an interface on that address's subnet; 3. otherwise any other interface, skipping loopback and `docker*` unless nothing else exists. `NCCL_SOCKET_IFNAME` bypasses that choice. Each entry is a **prefix**: `eth` matches `eth0`, `eth1` and so on; `=eth0` matches exactly `eth0`; `^docker` excludes a prefix; `^=docker0` excludes one name. The same selection serves the bootstrap, the RDMA transport's out-of-band connection setup and, when RDMA is absent, the socket data path. Gloo does not read it: a Gloo group needs `GLOO_SOCKET_IFNAME`. A cluster administrator can set the value once for every job in `/etc/nccl.conf`. ## Isolating the transports If every rank bootstraps on the right interface and the job still stalls once data moves: - `NCCL_IB_DISABLE=1` stops NCCL using InfiniBand or RoCE verbs and falls back to IP sockets. If the job now completes, the RDMA path is at fault: adapter selection, RoCE addressing, or the fabric. - `NCCL_P2P_DISABLE=1` disables direct GPU-to-GPU access over NVLink or PCIe, so intra-node traffic goes through shared host memory. Run one node alone with it to test that path. NCCL's guide lists both under its debugging variables: fine for experiments, not something to leave in a production launch script. ## Other setup causes worth ruling out - **A firewall between nodes.** NCCL opens TCP connections between ranks for the bootstrap; if they are blocked, the connect calls fail or time out. Since NCCL 2.24, `NCCL_SOCKET_RETRY_CNT` (default 34) retries a connection after a timeout, refusal or unreachable host, which is one reason a broken setup can sit for a while instead of failing at once. - **Mixed address families.** `NCCL_SOCKET_FAMILY=AF_INET` or `AF_INET6` forces one when nodes would otherwise pick different ones. - **A launch that never reached every node.** If a node's log file does not exist at all, the problem is upstream of NCCL. ## Mistakes interviewers listen for - Expecting `MASTER_ADDR` to steer NCCL's interface choice. - Pinning `NCCL_SOCKET_IFNAME` and assuming Gloo follows. - Leaving `NCCL_IB_DISABLE=1` in place and losing RDMA bandwidth for good. - Reading only rank 0's log when the misconfigured node is another one.

  • Every rank bootstraps on the right interface, yet the job still hangs once transfers start. What next?
    Split the transports. Rerun with `NCCL_IB_DISABLE=1` to force IP sockets between nodes; if it completes, the RDMA path (adapter selection, RoCE addressing, the fabric) is at fault. Run one node alone, then with `NCCL_P2P_DISABLE=1`, to test NVLink or PCIe. NCCL lists both as debugging variables, so remove them once the culprit is found.
  • Why can NCCL_SOCKET_IFNAME=eth0 still match more than one interface?
    Each entry is a prefix unless it starts with `=`, so `eth0` also matches any name beginning with `eth0`, such as a VLAN sub-interface. Use `=eth0` for exactly that interface, `^docker` to exclude a prefix and `^=docker0` to exclude one name. Any value bypasses NCCL's automatic selection.

  • NCCL_DEBUG=WARN is enough to see which interface NCCL chose.
  • NCCL reads MASTER_ADDR to decide which network interface to use.
  • Setting NCCL_SOCKET_IFNAME also fixes the interface Gloo uses.
  • NCCL_IB_DISABLE=1 is a safe permanent fix for a RoCE hang.
  • A hang in the first collective proves the model code is wrong.

Assumes NCCL 2.32: NCCL_DEBUG_SUBSYS defaults to INIT,BOOTSTRAP,ENV, and the ATTN level of NCCL_DEBUG is new in 2.32.