In multi-GPU LLM inference, how do tensor and pipeline parallelism differ?

## Why a model gets split at all A served model has to fit in GPU memory alongside its KV cache and activations, and it has to produce tokens fast enough. When one GPU cannot hold the weights, or when reading those weights once per token is too slow, the model itself is split across devices. For transformer inference there are two general ways to cut it, plus a third that only applies to mixture-of-experts models. ## Tensor parallelism: cut each layer sideways Tensor parallelism shards the matrices *inside* every layer. In the attention block the attention heads are divided among ranks — each GPU computes its own subset of heads — and in the MLP the up-projection is split by columns while the down-projection is split by rows. Each GPU therefore produces a *partial* result over the hidden dimension, and the ranks must sum those partials before the next sub-layer can proceed. That summation is an all-reduce, and there is one after the attention block and one after the MLP: roughly two collectives per transformer layer, per forward pass, for every token generated. What you buy: each GPU stores only 1/N of the weights and, because KV heads are split with the attention heads, only 1/N of the KV cache. Aggregate memory bandwidth also scales, and since single-stream decode is bandwidth-bound, tensor parallelism is the split that can genuinely lower per-token latency for one request. What you pay: N GPUs sit on the critical path of every layer, and the whole group only goes as fast as its slowest link. ## Pipeline parallelism: cut the stack crosswise Pipeline parallelism assigns each GPU a contiguous *stage* of layers. The only thing crossing a stage boundary is the hidden-state activation for the tokens being processed — a single tensor of roughly (tokens x hidden_size x dtype bytes), sent point-to-point. That is orders of magnitude less traffic than tensor parallelism's per-layer collectives, and it is why pipeline parallelism is the strategy that survives a slow or high-latency link such as an Ethernet fabric between nodes. The catch is the bubble. For one request the stages run strictly in sequence: stage 2 cannot start until stage 1 finishes, so a lone request sees no latency improvement (and a little extra, from the handoffs), while every other stage sits idle. Serving engines hide the bubble by keeping many requests in flight, so different stages work on different micro-batches at the same time. Pipeline parallelism is therefore a memory-and-throughput tool that assumes concurrency, not a latency tool. ## How they compose The two are orthogonal and are routinely combined: tensor-parallel within a node where the GPUs share a high-bandwidth fabric, pipeline-parallel across nodes. Total GPUs = TP degree x PP degree (x replicas). Both split memory, so both can be used purely to make a model fit; only tensor parallelism reliably makes a single request faster. ## Where the engines expose this The knob names belong to each engine and differ: vLLM takes `--tensor-parallel-size` and `--pipeline-parallel-size` at launch; TGI's `--num-shard` sets the tensor-parallel degree; TensorRT-LLM fixes the parallel layout when the engine is *built*, not when it is served, so changing the degree there means rebuilding the engine. Do not assume one spelling carries across engines. ## Common misreads "Pipeline parallelism makes generation faster" — it does not, for a single request. "Tensor parallelism is always better" — only where the interconnect can carry the collectives; over a weak link it can be slower than not sharding. "Splitting the model gives me more total memory for everything" — tensor parallelism does hand each rank a smaller weight and KV-cache slice, but it also imposes a collective per layer, and the group fails as a single unit: lose one rank and the whole replica is down.