skip to content

GPU Training Fundamentals

You will learn the hardware realities of training: why batch size is bounded by GPU memory, what mixed precision and loss scaling buy you, and when gradient accumulation or data parallelism is the right fix. Interviewers use these to separate practitioners who have trained real models from paper readers.

on this pageshow

explore

questions

page 2 of 2

A dense model moving from 4K- to 64K-token documents and a mixture-of-experts model are both tensor- and pipeline-split yet run out of memory — what do you split next for each, and what does it cost?

level: seniorimportance: nice to knowfreq 24%

basics

~20 s

Split by what grew. Long-sequence activations call for a context split: each device holds a slice of the tokens, and attention exchanges keys and values. Expert weights call for an expert split: whole experts per device, paid in per-layer all-to-alls.

open as a page

How do you combine tensor and pipeline parallelism when a checkpoint fits on no single device?

level: principalimportance: nice to knowfreq 30%

basics

~20 s

Match each strategy to the interconnect it needs. Tensor parallelism puts a blocking collective in every block, so keep it inside one high-bandwidth machine. Pipeline parallelism sends one small activation per stage boundary, so it tolerates slower links between machines.

open as a page

A training job OOMs the day before a deadline — in what order do you apply the memory levers?

level: principalimportance: nice to knowfreq 36%

basics

~20 s

Measure the true peak first, then go cheapest-to-riskiest: shrink the micro-batch, restore the effective batch with gradient accumulation, add activation recomputation, drop precision, then shard state across devices. Anything that changes what the model sees comes last.

open as a page

showing 31–33 of 33