Why does halving the training batch size cut device memory when the weights are unchanged?

## The two piles Everything that occupies device memory during a training step falls into one of two categories, and the whole answer follows from knowing which category each thing is in. **Pile one — model state.** This is memory whose size is a function of the parameter count and nothing else: - the **weights** themselves, one number per parameter; - the **gradients**, one number per parameter, because the backward pass produces exactly one partial derivative per weight; - the **optimizer state**, which is whatever the update rule carries between steps. Plain stochastic gradient descent carries nothing. SGD with momentum carries one velocity buffer per parameter. Adam and AdamW carry two: a first moment estimate and a second moment estimate. The critical point for this question is that **none of these depends on how many samples you push through**. A gradient is not accumulated per sample and stored per sample; the per-sample contributions are summed or averaged into a single gradient tensor as the backward pass runs. Sixteen samples and one sample produce a gradient tensor of exactly the same shape. **Pile two — activations.** On the forward pass, each layer computes an output tensor from its input. Most of those intermediate tensors have to be kept alive until the backward pass reaches that layer, because computing the gradient with respect to a layer's weights needs the values that flowed into or out of it. Those retained tensors are the activations. An activation tensor's shape starts with the batch dimension. A convolutional feature map is (batch, channels, height, width); a sequence model's hidden states are (batch, length, width). Every retained tensor carries that leading batch axis, so the sum over all of them is, to a very good approximation, `per_sample_bytes * batch_size`. Halve the batch and you halve it. ## Doing the arithmetic Suppose a run at batch 32 occupies 20 GB, and you know the model state is 12 GB. Then activations are about 8 GB, or 250 MB per sample. At batch 16 you expect roughly `12 + 4 = 16 GB` — a 20% saving, not a 50% saving. The mistake juniors make is assuming total memory is proportional to batch size; only one of the two piles is. The same arithmetic run the other way tells you the ceiling. As batch size goes to one, memory does not go to zero: it converges to `model_state + one_sample_activations`, plus small fixed overheads for temporary buffers a layer needs while computing. If that floor already exceeds the device, no amount of batch reduction rescues the run — the shape of a single sample has to change. ## Where the ratio comes from Which pile dominates depends on the architecture, and the two extremes look completely different: - A **large language-style stack or a very wide network trained on small inputs** is parameter-heavy. Model state can be tens of gigabytes while activations at a modest batch are a few. Here batch size is a weak lever. - A **convolutional network on high-resolution images or volumes** is activation-heavy. The parameter count may be tens of millions — under a gigabyte of state — while a single sample's retained feature maps run to several gigabytes because the early layers hold full-resolution tensors. Here batch size is the dominant lever, and the batch that fits may be a single-digit number. So "how much does batch size buy me" is not a fact about training in general; it is a fact about the ratio in your specific model, and a candidate who can name the ratio rather than a rule of thumb is answering at the right level. ## Secondary effects worth knowing A few details soften the clean linear picture. Some memory is allocated once and reused for scratch space regardless of batch size, so the activation share is linear plus a constant. Very small batches can also make the arithmetic on the device less efficient, so you pay in throughput for the memory you free. And normalization layers that compute statistics across the batch behave differently at small batch sizes — that is a statistical concern, not a memory one, but it is why shrinking the batch is not a free choice. The takeaway to carry into an interview: **parameter count sets a fixed floor; input shape and batch size set everything above it.** Being able to say which of your two numbers is which, and estimate both, is what the question is really testing.