Llama 70B won't fit one GPU — how do you choose between tensor parallelism, quantization, and a smaller model?

## Frame it as a decision, not a fitting exercise Llama 3.1 70B at fp16 is roughly 140 GB of weights plus ~320 KiB of KV cache per token. Nothing about that dictates an answer; the answer comes from three constraints you should state before touching a flag: the **quality floor** on your actual tasks, the **latency SLO** (time-to-first-token and inter-token latency), and the **cost per million tokens** at forecast volume. Candidates who jump straight to "shard it across four GPUs" have skipped the only interesting part. ## Lever 1: tensor parallelism Tensor parallelism splits individual weight matrices across GPUs — attention heads and MLP columns are partitioned, each GPU computes its slice, and the results are combined with an all-reduce at the end of each block. In vLLM this is `--tensor-parallel-size`; in TGI, `--num-shard`. What it buys: memory divides almost linearly, and because each GPU reads only its shard of the weights, per-token decode latency also improves — TP is the standard way to get a large model *fast*, not merely to make it fit. What it costs: two all-reduces per transformer layer, on every forward pass. On NVLink-connected GPUs inside one node this is cheap. Over PCIe, or worse across a network, the collective can dominate and you end up slower than a single-GPU quantized deployment. It also introduces failure coupling — one GPU lost takes the whole replica down — and the degree must divide the model's head counts, which for Llama's 8 KV heads means 1, 2, 4 or 8. Crossing nodes is a different regime: **pipeline parallelism** (`--pipeline-parallel-size`) splits layers rather than matrices, sending far less data between stages, so it tolerates slower links but adds pipeline bubbles and does not reduce per-request latency. The usual composition is TP within a node, PP across nodes — and "across nodes" is an admission that the model is too big for your hardware, with all the operational cost that implies. ## Lever 2: quantization Reducing weight precision cuts the weight budget proportionally and, just as importantly, frees VRAM that becomes KV cache — which is what actually sets concurrency. It can turn a two-GPU deployment into one, or make a 70B fit where it otherwise would not. The cost is accuracy, and the honest position is that it is workload-dependent: degradation that is invisible on chat may be fatal on structured extraction or code. Never accept a published benchmark as evidence for your workload; run your own evaluation set. Also confirm the serving stack supports the specific scheme on your hardware and that a quantized kernel path exists, or you will lose the speed you were buying. ## Lever 3: a smaller model The lever most often skipped, and most often correct. Llama 3.1 8B on a single GPU serves several times the throughput per dollar and eliminates the interconnect question entirely. Many production tasks — classification, routing, entity extraction, short summarisation, tool-argument filling — show little measurable gap once you have a real evaluation, especially with a good prompt or a task-specific fine-tune. A frequent architecture is a small model handling the bulk of traffic with escalation to a large one for the minority of hard cases, which usually beats running everything on the large model. ## The order of operations 1. **Build the evaluation first.** Without a task-specific quality bar, every subsequent comparison is opinion. 2. **Try the smaller model.** If it clears the bar, stop; you have removed the problem. 3. **Try quantized-large on the fewest GPUs that fit.** Re-run the evaluation, then load-test. 4. **Then multi-GPU full precision**, sized so tensor parallelism stays inside a node with fast interconnect. 5. **Price each survivor** per million tokens at forecast load, including idle GPU time — a large model at 20% utilisation is often more expensive per token than the hosted alternative. ## The strategic questions behind it Replica shape matters as much as fit: two independent TP=4 replicas give you rolling upgrades and survive a node failure; one TP=8 replica gives lower latency and a single point of failure. Workload isolation matters too — batch summarisation sharing a pool with interactive chat will violate the chat SLO regardless of parallelism strategy, so separate pools are frequently the right answer. And the framing question worth raising explicitly: does self-hosting 70B earn its keep at all? Self-hosting wins on data residency, per-token cost at genuinely high sustained volume, latency control, and freedom from vendor deprecation. It loses on engineering time, idle capacity, and the fact that the frontier moves. A principal-level answer names that tradeoff instead of assuming the fleet.