GPU Training Fundamentals

The input pipeline, not the model. The GPU idles waiting for batches while the host decodes and resizes images. Fixes are overlapping data preparation with compute, adding host-side parallelism, and pre-processing tiles into a cheaper stored form.

A 16-bit float's 5 exponent bits bottom out near 6e-5 for normal values and near 6e-8 once subnormals run out. A gradient smaller than that has no representation, so it stores as exactly zero and that weight stops moving.

Mixed-precision training runs the forward and backward passes in a 16-bit float while keeping a single-precision copy of the weights. Wins come from moving half the bytes and from matrix units that multiply 16-bit inputs at much higher throughput.

Weights, gradients and optimizer state are sized by the parameter count, not by batch size. Activations, the per-layer outputs held for the backward pass, exist once per sample, so halving the batch roughly halves that share.

Arithmetic intensity is the floating-point operations an op performs per byte it moves to and from device memory. Compare it with the device's peak-FLOP-rate divided by peak bandwidth: below that ridge point the op is bandwidth-bound, above it compute-bound.

Every device holds a full copy of the weights and processes its own slice of the batch. Local gradients are then averaged across all devices with a single all-reduce, so every replica applies the identical update and stays in sync.

Because each micro-batch's loss is usually already averaged over its own samples, so adding N of them gives N times the true full-batch gradient. The extra 1/N factor restores the mean over all samples and keeps the step size honest.

A mixed-precision step keeps the weights the optimizer updates, large reductions, the loss and normalization statistics in 32-bit. Those are the places where a tiny quantity is added to or accumulated with a much larger one, which 16-bit arithmetic destroys.

The bubble is the idle time while the pipeline fills and drains: with S stages, S-1 stage-slots are wasted at each end. Splitting the batch into M micro-batches makes the bubble fraction (S-1)/(M+S-1), so more micro-batches shrink it.

Reserved is what the caching allocator holds from the driver; allocated is what live tensors use. The gap is cached blocks of the wrong sizes, and a new tensor needs one contiguous block none of them can supply.

About 24 GB. Adam training in 32-bit floats costs 16 bytes per parameter - weight, gradient, and two moment estimates at 4 bytes each - so 1.5 billion parameters need roughly 24 GB before any activation exists.

Gradient accumulation runs several small micro-batches, adding their gradients into one buffer, and updates the weights only after the whole group. You get the update of a large batch while only one micro-batch's activations sit in memory at a time.

Place boundaries so retained memory balances: with n uniform layers in k segments it scales like k + n/k, smallest near k equal to the square root of n. When layers differ, rank candidates by bytes freed per recomputed operation.

No. Data parallelism puts a full copy of the weights, gradients and optimizer state on every device, so the model's own memory cost is unchanged. Only the activations shrink, because each replica sees a smaller micro-batch.

A 16-bit float keeps about 11 significand bits, so the next value above 1.0 is roughly 1.001. A 1e-7 update is far below half that gap, so the sum rounds back to 1.0 and the weight never moves.

Half-precision gradients can be too small to represent and collapse to zero. Multiplying the loss by a large constant scales every gradient up by that factor through the chain rule; the scale is divided back out before the optimizer step.

Split the first weight matrix by columns so each device computes its own slice of the hidden activations, then split the second by rows so each device produces a partial sum of the output. One all-reduce adds the partials.

The recomputed forward draws fresh random numbers, so the rebuilt activations differ from the ones that produced the loss and the gradients belong to a different sampled network. Fix it by capturing and restoring the generator state per segment.

Fusion removes the trips to device memory between the stages. Unfused, the activation tensor is written and re-read after each operation; fused, it is read once and written once. The arithmetic is identical — only the memory traffic shrinks.

Each replica normalizes with the mean and variance of its own four samples, not of the full effective batch. Gradient averaging cannot fix this, because the statistics are used in the forward pass. Synchronizing them across replicas restores single-device behaviour.

Clipping inside the loop rescales each micro-batch gradient separately, so the sum of clipped pieces is not the clipped full-batch gradient. Clip once after the final backward pass and before the step, so the threshold sees the real update.

bf16 carries the same exponent range as 32-bit float, so gradients that underflow in fp16 stay representable and no loss scaler is needed. The price is fewer mantissa bits, so 32-bit master weights and accumulation still matter.

Partition parameters across replicas so each device keeps only its slice of Adam's moments and master weights, updates that slice, then all-gathers the updated weights. State memory falls by the replica count at no extra communication.

Log memory per step. Steady growth with a constant increment means retention, usually a logged metric still attached to its graph. Flat memory with one tall spike means a peak: the longest sample, or the evaluation pass.

Budget two piles. Model state, parameter count times bytes per parameter, is often under a gigabyte here. Activations dominate: one 256-cubed feature map with 32 channels in half precision is about 1 GB, and several stay resident.

Not before you know where the missing throughput went. At 37 percent scaling efficiency each added device returns about a third of a device, and a step bound by gradient communication only gets worse with more replicas.

Check whether the bytes moved each step exceed what the host link can carry in the time available. Offloading pays only when a round trip is faster than recomputing the values and overlaps with compute.

A global norm sums squares across every parameter. Each square is finite, but the running total crosses the 16-bit ceiling of 65504 and saturates to infinity. The failure lives in the accumulator, not in any individual gradient.

Each micro-batch is normalized with statistics computed from its own samples alone, because normalization happens in the forward pass, before anything is accumulated. Accumulation enlarges the update, not the sample set a forward pass can see.