On a new multi-node cluster the first NCCL collective of a PyTorch job never returns; how do NCCL_DEBUG and NCCL_SOCKET_IFNAME help you find why?
answer
- the first call is also the setup
- make every rank say what it chose
- the Bootstrap and Using network lines
- ib-prefixed interfaces win by default
- a leading = for an exact name
basics
~10 sSet NCCL_DEBUG=INFO on every rank and read which interface each one bootstraps on and which network it chose; if NCCL picked an interface the other nodes cannot reach, pin the routable one with NCCL_SOCKET_IFNAME.
solid answer
~40 sThe first collective is where NCCL builds its communicator (unless a `device_id` made PyTorch create it eagerly), so a hang there is usually a setup problem. Set `NCCL_DEBUG=INFO`, add `NET` to `NCCL_DEBUG_SUBSYS` (default `INIT,BOOTSTRAP,ENV` in NCCL 2.32), and write per-rank logs with `NCCL_DEBUG_FILE` and its `%h.%p` pattern. Compare the `Bootstrap: Using <interface>:<address>` line and the `Using network IB` or `Using network Socket` line across nodes. With `NCCL_SOCKET_IFNAME` unset, NCCL prefers interfaces whose names start with `ib`, then falls back to any other except loopback and docker ones, which can be a management or virtual interface the other nodes cannot reach. Pin the data interface, for example `NCCL_SOCKET_IFNAME="=ens1f0"`. If setup then completes but transfers stall, `NCCL_IB_DISABLE=1` and `NCCL_P2P_DISABLE=1` isolate the network and intra-node paths for a test run only.
code
bash · 10 lines# On every node, before the ranks start
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,BOOTSTRAP,ENV,NET
export NCCL_DEBUG_FILE=/shared/nccl-logs/nccl.%h.%p
# After a failed run: which interface and network did each rank pick?
grep -hE "Bootstrap: Using|Using network" /shared/nccl-logs/nccl.*
# Once the logs show the wrong interface, pin the data interface exactly
export NCCL_SOCKET_IFNAME="=ens1f0"go deeper
Recall the two variables: NCCL_DEBUG=INFO makes NCCL explain its choices, and NCCL_SOCKET_IFNAME tells it which network interface to use.
Explain why the first collective is where NCCL sets up, read the Bootstrap and Using network lines, and describe NCCL's automatic interface order and the = and ^ syntax for pinning one.
Show a method: per-rank log files, compare advertised addresses across nodes, isolate transports with the disable toggles, and remove every toggle once the cause is fixed.
Push the fix into the platform, for example a cluster-wide /etc/nccl.conf with the right interface names, so no training team rediscovers it on the next node pool.
## Why the first collective is where it hangs NCCL is not a launcher: it relies on the application's processes and on a CPU-side **bootstrap** over TCP to find its peers. PyTorch creates the NCCL **communicator** lazily, at the first collective on a process group, unless `init_process_group` was given a `device_id`, in which case it is created immediately. So "the first all-reduce never returns" usually means one of three things: 1. The **bootstrap** cannot connect, because a rank advertised an address the others cannot reach. 2. The bootstrap succeeds but the **data transport** (RDMA adapters, peer-to-peer inside the node) cannot connect. 3. The ranks really are inside different collectives, a code problem rather than a network one. The first two leave fingerprints in NCCL's own log; the job eventually dies on PyTorch's process-group timeout either way. ## Make every rank report what it chose - `NCCL_DEBUG=INFO` prints NCCL's debug information. Unset, NCCL prints nothing; `WARN` prints only errors when NCCL calls fail, which a silent hang may never produce. NCCL 2.32 adds an `ATTN` level, `WARN` plus notices. - `NCCL_DEBUG_SUBSYS` filters the INFO output. Its default in NCCL 2.32 is `INIT,BOOTSTRAP,ENV`; add `NET` for transport detail and `GRAPH` for topology detection. - `NCCL_DEBUG_FILE` writes the log to a file; `%h` and `%p` expand to host name and process ID, so 32 ranks do not overwrite one file. ## Reading the log | Line in the INFO output | What it tells you | |---|---| | `NCCL_SOCKET_IFNAME set by environment to ...` | The interface filter actually reached this rank | | `Bootstrap: Using <if>:<address>` | The interface and address this rank advertised to its peers | | `Using network IB` or `Using network Socket` | Whether inter-node data goes over RDMA verbs or IP sockets | | `NET/IB : Using [0]mlx5_0:1/RoCE ...; OOB <if>:<address>` | The RDMA adapters found, their link layer, and the socket interface used for connection setup | | `Bootstrap : no socket interface found` | No interface survived the selection or the filter at all | If the logs stop right after the `Bootstrap` line, compare the advertised addresses: one node offering a management, bridge or link-local address is the classic new-cluster hang. ## How NCCL picks an interface, and how to pin one With `NCCL_SOCKET_IFNAME` unset, NCCL 2.32 selects automatically: 1. interfaces whose names start with `ib`; 2. otherwise, if `NCCL_COMM_ID` is set, an interface on that address's subnet; 3. otherwise any other interface, skipping loopback and `docker*` unless nothing else exists. `NCCL_SOCKET_IFNAME` bypasses that choice. Each entry is a **prefix**: `eth` matches `eth0`, `eth1` and so on; `=eth0` matches exactly `eth0`; `^docker` excludes a prefix; `^=docker0` excludes one name. The same selection serves the bootstrap, the RDMA transport's out-of-band connection setup and, when RDMA is absent, the socket data path. Gloo does not read it: a Gloo group needs `GLOO_SOCKET_IFNAME`. A cluster administrator can set the value once for every job in `/etc/nccl.conf`. ## Isolating the transports If every rank bootstraps on the right interface and the job still stalls once data moves: - `NCCL_IB_DISABLE=1` stops NCCL using InfiniBand or RoCE verbs and falls back to IP sockets. If the job now completes, the RDMA path is at fault: adapter selection, RoCE addressing, or the fabric. - `NCCL_P2P_DISABLE=1` disables direct GPU-to-GPU access over NVLink or PCIe, so intra-node traffic goes through shared host memory. Run one node alone with it to test that path. NCCL's guide lists both under its debugging variables: fine for experiments, not something to leave in a production launch script. ## Other setup causes worth ruling out - **A firewall between nodes.** NCCL opens TCP connections between ranks for the bootstrap; if they are blocked, the connect calls fail or time out. Since NCCL 2.24, `NCCL_SOCKET_RETRY_CNT` (default 34) retries a connection after a timeout, refusal or unreachable host, which is one reason a broken setup can sit for a while instead of failing at once. - **Mixed address families.** `NCCL_SOCKET_FAMILY=AF_INET` or `AF_INET6` forces one when nodes would otherwise pick different ones. - **A launch that never reached every node.** If a node's log file does not exist at all, the problem is upstream of NCCL. ## Mistakes interviewers listen for - Expecting `MASTER_ADDR` to steer NCCL's interface choice. - Pinning `NCCL_SOCKET_IFNAME` and assuming Gloo follows. - Leaving `NCCL_IB_DISABLE=1` in place and losing RDMA bandwidth for good. - Reading only rank 0's log when the misconfigured node is another one.
- Every rank bootstraps on the right interface, yet the job still hangs once transfers start. What next?Split the transports. Rerun with `NCCL_IB_DISABLE=1` to force IP sockets between nodes; if it completes, the RDMA path (adapter selection, RoCE addressing, the fabric) is at fault. Run one node alone, then with `NCCL_P2P_DISABLE=1`, to test NVLink or PCIe. NCCL lists both as debugging variables, so remove them once the culprit is found.
- Why can NCCL_SOCKET_IFNAME=eth0 still match more than one interface?Each entry is a prefix unless it starts with `=`, so `eth0` also matches any name beginning with `eth0`, such as a VLAN sub-interface. Use `=eth0` for exactly that interface, `^docker` to exclude a prefix and `^=docker0` to exclude one name. Any value bypasses NCCL's automatic selection.
- NCCL_DEBUG=WARN is enough to see which interface NCCL chose.
- NCCL reads MASTER_ADDR to decide which network interface to use.
- Setting NCCL_SOCKET_IFNAME also fixes the interface Gloo uses.
- NCCL_IB_DISABLE=1 is a safe permanent fix for a RoCE hang.
- A hang in the first collective proves the model code is wrong.
Assumes NCCL 2.32: NCCL_DEBUG_SUBSYS defaults to INIT,BOOTSTRAP,ENV, and the ATTN level of NCCL_DEBUG is new in 2.32.