Training Hyperparameters

## Effective batch is the number that matters Optimization does not care how a batch was assembled — it cares how many examples contributed to the gradient before the weights moved. That quantity is the **effective batch size**: effective_batch = micro_batch × accumulation_steps × devices The micro-batch is what one forward/backward pass holds in memory at once. The accumulation steps are how many such passes contribute to a single optimizer update. On multi-GPU runs, data-parallel replicas multiply it again. When a paper or a recipe quotes "batch size 64", it means the effective batch, and reproducing the run means matching that product, not one of its factors. ## The mechanic On each micro-batch you run the forward pass, compute the loss, and run the backward pass, which *adds* into the gradient buffers rather than replacing them. You do not step the optimizer. After N micro-batches you take one optimizer step and zero the gradients. Mathematically this reproduces the gradient of a single batch of N × micro-batch examples — subject to the normalization caveat below — while never holding more than one micro-batch of activations in memory. ## Why memory, not arithmetic, is the constraint During a backward pass the framework must retain intermediate activations from the forward pass. That storage scales roughly with micro-batch size × sequence length (attention adds further terms), and on a long-context fine-tune it dominates everything else. Model weights, gradients and optimizer state, by contrast, are fixed regardless of batch size. This is why the practical tuning loop is: pick the sequence length the task needs, find the largest micro-batch that does not OOM at that length, then set accumulation steps to reach the effective batch you want. A run at 4,096-token sequences might only fit micro-batch 2 on an 80 GB card, so an effective batch of 16 is 2 × 8. Halving the sequence length often lets micro-batch double and accumulation halve, for the same effective batch and better throughput. ## What accumulation does not buy you It is a memory trick, not a speed trick. You still perform the same amount of compute for the same number of examples, and small micro-batches under-utilise the GPU, so accumulation is usually *slower* per example than a genuine large batch would be on hardware that could hold it. It also does not shrink optimizer state or model weights — if the model plus optimizer does not fit, accumulation will not save you; quantization of the frozen base or adapter-only training will. ## The normalization pitfall The backward pass sums gradients across micro-batches, so the loss must be scaled down or the effective step is N times too large. The naive fix is to divide each micro-batch loss by N. That is exact only when every micro-batch contributes the same number of loss-bearing tokens. With variable-length examples, or when loss is computed on completion tokens only, micro-batches carry different token counts, and dividing by N over-weights the short ones. The correct normalization divides by the *total* loss-bearing token count across the accumulation window. This was a real, widely-reproduced bug in popular training stacks, and it is a good senior-level detail: it silently changes what you are optimizing without ever throwing an error. ## How large should the effective batch be Larger batches give lower-variance gradient estimates and more stable training, and they let you raise the learning rate — up to a point, beyond which extra examples per step stop buying anything and simply cost compute. For adapter-based fine-tuning there is a sharper limit: LoRA-style training degrades at large batch sizes noticeably earlier than full fine-tuning does, and increasing adapter capacity does not rescue it. The practical guidance that emerged from systematic studies is to keep the **effective batch under roughly 32** for adapter runs. That is a genuinely useful interview answer because it inverts the usual instinct that a bigger batch is always safer. ## Interactions to state out loud - **Learning rate.** Changing the effective batch changes what peak rate is appropriate; a rate found at effective batch 8 is not automatically right at 64. - **Steps per epoch.** Steps = examples ÷ effective batch. Raising the effective batch shortens the run in steps, which interacts with the schedule length and with how often you evaluate. - **Reproducibility.** Two runs match only if the *product* matches; 4 × 4 on one GPU and 2 × 4 on two GPUs are the same effective batch and should train alike, modulo data ordering.