Why does full fine-tuning of an 8B Llama need far more VRAM than its weights?

## Counting the terms People size training runs by checkpoint size and are then surprised by an out-of-memory error. Enumerate what a standard mixed-precision AdamW step actually keeps resident, per parameter: - **Model weights in compute precision** — bf16, 2 bytes. - **Gradients** — usually bf16, 2 bytes; some setups accumulate in fp32 at 4 bytes. - **fp32 master weights** — 4 bytes. Mixed precision keeps a full-precision copy so that tiny updates are not lost to bf16 rounding. - **AdamW first moment (`exp_avg`)** — fp32, 4 bytes. - **AdamW second moment (`exp_avg_sq`)** — fp32, 4 bytes. That totals 16 bytes per parameter. For an 8B Llama: 8e9 x 16 = ~128 GB, against a 16 GB bf16 checkpoint. The optimizer alone is four times the model. ## Activations: the term that varies Everything above is fixed by parameter count. Activation memory is not — it scales with batch size, sequence length and hidden width, because the backward pass needs the intermediate tensors produced by the forward pass. Attention and the MLP block both materialise large intermediates per layer. This is why the same configuration trains fine at 1,024 tokens and OOMs at 8,192, and why activation memory, not weights, is the binding constraint on long-context fine-tuning. ## The mitigations, in the order you reach for them **Gradient checkpointing.** Store only a few activation checkpoints per layer boundary and recompute the rest during the backward pass. Cuts activation memory dramatically at roughly 20-40% extra compute. Almost always the first thing you turn on. **Sharding across GPUs.** PyTorch FSDP and DeepSpeed ZeRO-3 partition parameters, gradients and optimizer state across ranks, so each GPU holds 1/N of the 16-bytes-per-parameter bill and gathers full weights only for the layer it is currently computing. This turns full fine-tuning of an 8B model into a routine multi-GPU job rather than an impossible single-GPU one. The cost is communication volume — all-gather on the forward pass, reduce-scatter on the backward — which makes interconnect bandwidth the limiting factor. **Offload.** ZeRO-Offload style techniques push optimizer state and sometimes parameters to host RAM or NVMe. It works, and it is slow; treat it as the option that lets a job finish rather than the one that makes it fast. **Cheaper optimizers.** 8-bit Adam roughly halves the moment storage. Optimizers that avoid a full second moment reduce it further. Each trades some convergence robustness for memory. **Smaller micro-batches with gradient accumulation.** Keeps the effective batch size while shrinking the resident activation footprint, at the cost of more steps per update. ## Why this arithmetic is the argument for adapters LoRA freezes the base, so there are no base gradients, no base optimizer moments and no fp32 master copy for the base — those three terms, 14 of the 16 bytes, disappear. What remains is 2 bytes per frozen parameter for the weights plus a full training bill for the ~0.1-1% of parameters that are adapters, plus activations. QLoRA then attacks the last remaining base term by holding the frozen weights in 4 bits. Understanding the 16-bytes-per-parameter breakdown is what makes the adapter approach obviously correct rather than merely fashionable. ## When full fine-tuning is still worth it It is not obsolete. Full fine-tuning is the right call when you are adapting a small Llama (1B-8B) on a large, high-quality corpus and want the model to genuinely internalise a new domain or language rather than acquire a behavioural veneer; when you are changing the tokenizer or adding many new tokens; or when you are doing continued pretraining rather than instruction tuning. It also removes the adapter-management burden at serving time. The price, besides hardware, is catastrophic forgetting — a fully fine-tuned model can lose general capability that an adapter would have left intact — so you evaluate on general benchmarks, not only on your task. ## Estimating before you launch A usable estimate: `bytes ≈ 16 x params` for the fixed terms, then add activations measured empirically at a small sequence length and scaled linearly in tokens per micro-batch. Compare against `num_gpus x memory_per_gpu x 0.85` to leave allocator headroom. If the fixed term alone does not fit across the cluster, no amount of batch-size tuning will save the run, and the honest answer is sharding, offload, or an adapter method.