What does QLoRA change about a LoRA run, and why does it fit on one GPU?

## The memory term LoRA does not remove Standard LoRA removes gradients, optimizer moments and master copies for the frozen parameters. What it cannot remove is the frozen model itself: every weight must be resident to run the forward pass. At 16 bits per parameter that is roughly 2 GB per billion parameters — about 14 GB for a 7B model, 26 GB for a 13B, 140 GB for a 70B. Once optimizer memory is gone, the base weights become the dominant term, and for larger models they alone exceed a single accelerator's memory. QLoRA is the direct answer: hold the frozen base at 4 bits instead of 16. ## The mechanism Three pieces do the work. **A 4-bit frozen base.** The pretrained weights are quantized once, at load time, into a 4-bit representation and never updated. During training, each weight block is dequantized back to a compute precision immediately before its matrix multiplication and discarded afterwards, so the high-precision copy exists only transiently for the block being used. Weight memory drops by roughly a factor of four. **Higher-precision adapters.** The LoRA matrices stay in bf16 and are the only trainable tensors. Gradients flow backward *through* the quantized frozen layers to reach them — the frozen weights participate in the backward computation but receive no update. **Memory hygiene around the edges.** Quantization stores per-block scaling constants, which themselves consume memory; quantizing those constants in turn reclaims a further slice. Separately, optimizer state can be paged out to host memory so that transient allocation spikes — long sequences, an unlucky batch — do not abort a run that would otherwise fit. ## Why freezing makes low precision tolerable This is the conceptual point worth making in an interview. Quantization error in a *trainable* weight compounds: you quantize, update, requantize, and the rounding interacts with the optimization trajectory. In QLoRA the base is frozen, so its quantization error is introduced exactly once, at load, and stays fixed for the whole run. The trainable path — the adapters — is at full training precision throughout. The optimization problem is therefore a clean bf16 problem over a small parameter set, applied on top of a slightly perturbed but static function. ## What it costs **Step time.** Dequantizing blocks on every forward and backward pass is real work that a bf16 run does not do. Expect meaningfully slower steps than plain LoRA on the same hardware — you are trading throughput for the ability to run at all. **Merge friction.** Adding the scaled adapter product back into a 4-bit base is awkward: you would be merging a high-precision delta into a low-precision weight and then requantizing, which introduces a second round of error. The clean path is to merge into the original 16-bit weights and, if a quantized deployment artifact is wanted, quantize afterwards. **Operational coupling.** The adapter was trained against a specific quantized view of the base. Serving it against a differently quantized base is a configuration mismatch waiting to surprise you, so the quantization setup belongs in the adapter's recorded metadata alongside rank and scaling. ## When to reach for it QLoRA is a memory-constrained choice, not a default. If the run already fits in bf16 with room for your sequence length and batch, plain LoRA is faster and simpler. Reach for the 4-bit base when the base weights are what is blocking you: a large model on a single card, a shared cluster where you can claim only one device, or a development loop on a workstation. The decision is a capacity decision first — does the run fit at all — and everything else follows from that. ## The economics, concretely At roughly 2 bytes per parameter in bf16 and roughly 0.5 bytes plus small per-block overhead in 4-bit, a 7B base moves from about 14 GB to under 4 GB, a 13B from about 26 GB to around 7 GB, and a 70B from about 140 GB to roughly 35-40 GB. Add adapter parameters, their optimizer state, activations and workspace, and the qualitative shift is that models which needed several accelerators for adapter training become single-device jobs. That accessibility shift, more than any quality argument, is why the technique became the standard fallback whenever memory is the binding constraint.