How do you split one large LLM across two 8-GPU nodes for serving?

## Read the topology first Multi-node serving is a placement problem, and the placement is dictated by where bandwidth drops. Inside a modern GPU node, peer bandwidth is in the hundreds of GB/s with microsecond-scale latency. Between nodes, even a good fabric is far behind that, and a commodity Ethernet fabric is not close. Every layout decision follows from matching each parallelism strategy to the link it can tolerate. ## The default answer Tensor parallelism generates roughly two all-reduces per transformer layer per token — the heaviest, most latency-sensitive traffic in inference. It goes inside the node: TP = 8, the full node. Pipeline parallelism sends one activation tensor per stage boundary — a slab of size (tokens x hidden_size x dtype bytes), often well under a megabyte, point-to-point, and not on a per-layer cadence. It goes across the node boundary: PP = 2, node 0 running the first half of the layers, node 1 the second half. So 16 GPUs become TP=8 x PP=2. This is the layout to state first, because it is right in the overwhelming majority of cases. ## Sanity checks before you build it **1. Do you need 16 GPUs at all?** A 16-GPU single instance is the most expensive, most fragile serving unit you can construct: one failed GPU anywhere kills it, deploys are all-or-nothing, and scheduling requires two whole nodes simultaneously. If quantizing the weights lets the model fit in one node, two independent 8-GPU replicas give you more throughput, half the blast radius and vastly simpler operations. The senior answer always considers not sharding across nodes. **2. Do you have the concurrency?** Pipeline parallelism only pays when enough requests are in flight to keep both stages busy; otherwise one node is idle while the other works, and single-request latency is slightly *worse* than it would be on a hypothetical single node. High-throughput batch workloads fill a pipeline happily. A low-QPS interactive endpoint does not. **3. What is the fabric?** Cross-node tensor parallelism is not categorically impossible — with a high-end RDMA fabric and GPUDirect it can be made to work — but the collectives are being asked to cross a link that is much slower than the one they were designed for, and the result is usually worse than the pipeline layout on the same hardware. On Ethernet without RDMA it is not a serious option. ## Operational consequences to raise unprompted - **Failure domain.** All 16 ranks form one process group. A hung rank presents as a stalled collective, not a clean crash, so you need liveness checks with timeouts rather than relying on the process to exit. - **Startup.** Two nodes must both pull tens of gigabytes of weights and rendezvous before the server is ready. Cold start is measured in minutes, which interacts badly with aggressive autoscaling. - **Scheduling.** You now need gang scheduling: sixteen GPUs across two specific nodes, available at once. Partial placement is a deadlock waiting to happen. - **Network isolation.** Noisy neighbours on a shared fabric become your latency problem, and inter-node traffic should be on a dedicated high-speed interface, not the general-purpose one. ## How the engines handle it Multi-node support differs sharply and is worth naming rather than assuming. vLLM supports multi-node serving with a distributed executor backend — Ray or multiprocessing — coordinating ranks across machines, and exposes the two degrees as `--tensor-parallel-size` and `--pipeline-parallel-size`. TensorRT-LLM bakes the parallel layout into the compiled engine, so the split is decided at build time and changing it means rebuilding. Whichever stack you use, the layout must be identical on every node and the weights must be reachable from both. ## What a weak answer looks like "Set the tensor-parallel size to 16." It is a legal configuration and it will start, which is what makes it a good trap — it just runs badly, because you have placed the per-layer collectives on the slowest link in the system. The strong answer names the boundary, matches the strategy to the link, and then argues about whether the split was necessary at all.