In LoRA fine-tuning of Llama, what do rank r and lora_alpha control?

## The problem LoRA solves A Llama checkpoint is a stack of transformer blocks whose parameters are large dense matrices. Updating every one of them (full fine-tuning) means storing a gradient and optimizer state for each parameter, which is why an 8B model needs well over a hundred gigabytes of accelerator memory to train conventionally. LoRA (Low-Rank Adaptation) sidesteps this: it leaves every original weight frozen and learns a small additive correction instead. ## The math, in plain terms For a frozen linear layer with weight matrix `W` of shape `d_out x d_in`, LoRA adds two small matrices: `A` of shape `r x d_in` and `B` of shape `d_out x r`, where `r` (the rank) is small — 8, 16, 32, 64. The layer now computes `y = Wx + (alpha/r) * B(Ax)`. `A` is initialised randomly and `B` is initialised to zeros, so at step zero `B*A` is exactly zero and the adapted model is bit-identical to the base model. Training only updates `A` and `B`. The parameter saving is dramatic. A 4096 x 4096 projection has ~16.8M parameters; its rank-16 LoRA pair has `16*4096 + 4096*16` = ~131K, under 1%. Across a whole Llama model, LoRA typically trains 0.1%-1% of the parameters. ## What rank r actually buys Rank is a capacity knob. The product `B*A` can only ever have rank r, so r caps how many independent directions the adaptation can move the weights in. Low ranks (4-8) suit narrow stylistic or format adaptation on a few thousand examples. Higher ranks (32-128) give the adapter more room for a genuinely different task or a large dataset, at the cost of more trainable parameters, more optimizer memory, and a greater chance of overfitting a small dataset. The empirical finding that has held up well is that **breadth beats depth**: adapting more module types at r=16 usually outperforms adapting only the query and value projections at r=128. ## What lora_alpha actually buys `lora_alpha` is not a learning rate and not a trainable parameter — it is a constant multiplier on the adapter's output. PEFT divides it by r, so the effective scale is `alpha/r`. That division exists so that r and the update magnitude are decoupled: if you tune at r=8, alpha=16 (scale 2.0) and then raise r to 32, keeping alpha at 2x r (alpha=64) keeps the scale at 2.0 and you only change capacity, not strength. If you instead hold alpha fixed while raising r, the effective update shrinks and your carefully tuned learning rate is now wrong. A variant, rank-stabilised LoRA (`use_rslora=True` in PEFT), scales by `alpha/sqrt(r)` instead, which behaves better at high ranks. ## Choosing target_modules on Llama Llama's decoder blocks expose these linear layers by name: the attention projections `q_proj`, `k_proj`, `v_proj`, `o_proj`, and the SwiGLU MLP projections `gate_proj`, `up_proj`, `down_proj`. A conservative configuration targets the four attention projections; a stronger one targets all seven. Adapting the embedding and output head is possible but expensive on Llama because the vocabulary is large, and it is only necessary if you added new special tokens. PEFT also accepts `target_modules="all-linear"` as a shorthand for every linear layer outside the head. ## Practical defaults and their consequences A workable starting point for instruction-style adaptation of an 8B Llama is r=16, lora_alpha=32, lora_dropout=0.05, all seven projections, learning rate around 1e-4 to 2e-4 — an order of magnitude higher than full fine-tuning uses, because you are training a small randomly-initialised module rather than nudging a pretrained one. Train for one to three epochs and watch a held-out split; LoRA overfits small datasets quickly at high rank. ## What LoRA does not change The base weights never move, so you keep the original checkpoint intact and ship a few tens of megabytes of adapter per task instead of a full model copy. Inference memory for the base model is unchanged — LoRA is a training-time saving plus a distribution convenience, not a way to make a 70B model fit where it did not fit before. And because the adapter is additive on top of a frozen model, catastrophic forgetting is far milder than with full fine-tuning, though it is not zero.