skip to content

Sharding & Model-Parallel Engines

FSDP2's fully_shard and DeepSpeed ZeRO shard weights, gradients and optimizer state; Megatron-Core sets tensor, pipeline, context and expert sizes. LLM-training rounds ask for the memory math.

on this pageshow

questions

6

In DeepSpeed ZeRO, what do stages 1, 2 and 3 each partition, and which stage fits a 70B model on 256 80 GB GPUs?

level: middleimportance: must knowfreq 30%

answer

  1. one integer in the JSON config
  2. optimizer state, then gradients, then weights
  3. zero_optimization.stage
  4. divide by the data-parallel size
  5. offload_param needs stage 3

basics

~20 s

Stage 1 partitions optimizer states across data-parallel ranks, stage 2 adds the gradients and stage 3 adds the 16-bit weights. For 70B parameters on 256 GPUs only stage 3 fits: about 4.4 GB of model state per GPU.

solid answer

~50 s

`zero_optimization.stage` picks what DeepSpeed partitions across the data-parallel ranks: stage 1 the optimizer states (fp32 master weights and Adam moments), stage 2 also the reduced 16-bit gradients, stage 3 also the 16-bit weights, which it gathers layer by layer in forward and backward. Taking the usual 2 + 2 + 12 bytes per parameter for bf16 Adam, a 70B model on 256 ranks still keeps about 283 GB per GPU under stage 1 and 144 GB under stage 2, so neither fits in 80 GB; stage 3 leaves about 4.4 GB of model state per GPU and the rest for activations. Offload is the next lever: `offload_optimizer` (valid with stages 1-3) moves optimizer state and the update to CPU or NVMe, `offload_param` (stage 3 only) moves the weights too, both paid for in PCIe traffic and a slower step.

code

json · 12 lines
json
{
  "train_batch_size": 1024,
  "train_micro_batch_size_per_gpu": 1,
  "gradient_accumulation_steps": 4,
  "bf16": { "enabled": true },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "reduce_bucket_size": 500000000
  }
}

go deeper

for a junior

Recall the ladder: stage 1 partitions optimizer states, stage 2 adds gradients, stage 3 adds the weights, chosen by zero_optimization.stage in the DeepSpeed JSON config.

for a middle

Explain what each stage leaves whole on every rank, then divide the 16 bytes per parameter by the data-parallel size to show which stage fits a stated model on a stated cluster.

for a senior

Show judgment beyond the stage: when offload_optimizer or offload_param earns its PCIe cost, how overlap_comm and bucket sizes move step time, and how the batch triple must change when the GPU count does.

for a principal

Weigh ZeRO-3's per-layer gathers across hundreds of ranks against keeping some model parallelism inside the node, and decide where the memory budget comes from before buying host memory or NVMe for offload.

## What `zero_optimization.stage` controls DeepSpeed's **ZeRO** (Zero Redundancy Optimizer) removes the copies of training state that plain data parallelism keeps on every rank. Which copies it removes is set by one integer in the DeepSpeed JSON config, `zero_optimization.stage`. In DeepSpeed 0.19 the field defaults to `0`, which means ZeRO is off. Every stage partitions across the **data-parallel ranks** — all 256 GPUs in a pure data-parallel run, not only the eight inside one node. | Stage | Partitioned across ranks | Still whole on every rank | What it adds to the step | |---|---|---|---| | 0 | nothing | weights, gradients, optimizer states | gradient averaging only | | 1 | optimizer states (fp32 master weights, Adam moments) | 16-bit weights and gradients | updated weight partitions gathered after the optimizer step | | 2 | optimizer states and reduced 16-bit gradients | 16-bit weights | gradients reduced to the rank owning each partition; updated weights gathered after the step | | 3 | optimizer states, gradients and 16-bit weights | nothing | weights gathered layer by layer in forward and again in backward | Stage 3's extra cost is the one to remember. Because no rank holds the whole model, each layer's weights are gathered just before that layer runs, in forward and again in backward, and released afterwards. How those all-gathers and reduce-scatters perform on the wire belongs to the collectives material; what matters for this engine is **when** ZeRO issues them and **what** each stage leaves resident. ## The arithmetic for a 70B model on 256 GPUs The per-parameter byte count itself is foundation arithmetic: bf16 training with Adam keeps about 2 bytes of 16-bit weight, 2 bytes of 16-bit gradient and 12 bytes of fp32 master weight plus two moments — 16 bytes in all. ZeRO decides which of those terms get divided by the data-parallel size, here 256: 1. **Stage 1:** 2 + 2 + 12/256 ≈ 4.05 bytes per parameter, about **283 GB** per GPU for 70B parameters. 2. **Stage 2:** 2 + (2 + 12)/256 ≈ 2.05 bytes per parameter, about **144 GB** per GPU. 3. **Stage 3:** (2 + 2 + 12)/256 = 0.0625 bytes per parameter, about **4.4 GB** per GPU. On 80 GB parts only stage 3 fits. It leaves most of each GPU for activations and for the temporary full copy of whichever layer is currently gathered. No ZeRO stage divides activations; they are a separate budget handled by recomputation and the parallelism choices. ## Offload: `offload_optimizer` and `offload_param` - `offload_optimizer` is documented in DeepSpeed 0.19 as valid with stages 1, 2 and 3. It keeps the optimizer states on the CPU or on NVMe (`"device": "cpu"` or `"nvme"`) and runs the optimizer step on the CPU. - `offload_param` is valid only with stage 3. It moves the partitioned 16-bit weights off the GPU as well and fetches them back before use; stages 1 and 2 keep full weights on every GPU by design, so there is nothing partitioned to move. - Both blocks default `pin_memory` to `true`, which speeds host-device copies but locks that host memory. - The bill: every step crosses PCIe (gradients out, updated values back) and the CPU optimizer step is slower than the GPU one. For this run stage 3 already fits, so offload is a lever for a smaller cluster or longer sequences, not a default. ## Other keys whose behaviour follows the stage - `overlap_comm` defaults to `None` in DeepSpeed 0.19 — a dynamic default that resolves to `true` at stage 3 and `false` at stages 1 and 2. Set it explicitly to overlap gradient reduction with backward at stage 2. - `reduce_bucket_size` and `allgather_bucket_size` both default to `5e8` elements. Larger buckets mean fewer, bigger collectives and more temporary memory. - `contiguous_gradients` defaults to `true`: gradients are copied into one contiguous buffer as they are produced, which avoids fragmentation during backward. ## The batch triple and the `bf16` block DeepSpeed checks at start-up that `train_batch_size` equals `train_micro_batch_size_per_gpu` × `gradient_accumulation_steps` × the data-parallel size, and stops when it does not; give any two and it infers the third. With 1 sample per GPU, 4 accumulation steps and 256 data-parallel ranks, one optimizer step sees 1,024 samples. The `bf16` block's `"enabled": true` turns on bfloat16 training. By default the master weights and optimizer states stay fp32; the `bf16_master_weights_and_grads` and `bf16_optimizer_states` keys move them to bf16 when you accept that precision trade. Why bf16 needs no loss scaling is the mixed-precision foundation's subject, not the engine's. ## Mistakes interviewers listen for - Saying stage 1 or 2 removes the weights from each GPU — only stage 3 partitions them. - Putting `offload_param` into a stage-2 config. - Reading `train_batch_size` as the per-GPU batch. - Forgetting that stage 3 gathers every layer twice per step, trading communication for memory. - Believing ZeRO-3 shards only within a node; it partitions across the whole data-parallel group.

  • When would you add `offload_optimizer` to the 70B ZeRO-3 config, and what does it cost?
    When stage 3 alone leaves too little room for activations, for example a longer sequence or fewer GPUs. With `"device": "cpu"` the fp32 master weights and Adam moments live in host memory and the update runs on the CPU: about 12 × 70B / 256 ≈ 3.3 GB of host memory per GPU, roughly 26 GB per eight-GPU node. The price is PCIe traffic every step and a CPU optimizer step slower than the GPU one.
  • A DeepSpeed config sets `train_batch_size` 1024, micro-batch 1 and accumulation 4, and the job stops at start-up on 128 GPUs. Why?
    DeepSpeed asserts that `train_batch_size` equals `train_micro_batch_size_per_gpu` × `gradient_accumulation_steps` × the data-parallel size. On 128 data-parallel ranks 1 × 4 × 128 = 512, not 1024, so the batch check fails. Set accumulation to 8 to keep 1,024 samples per step, or give only two of the three keys and let DeepSpeed infer the third.
  • Can you offload the weights to CPU from a ZeRO stage-2 config?
    No. DeepSpeed 0.19 documents `offload_param` as valid only with stage 3: stages 1 and 2 keep the full 16-bit weights on every GPU by design, so there is no partitioned weight store to move. To push weights off the GPU, move to stage 3 first; `offload_optimizer` is the offload that works with stages 1, 2 and 3.

  • ZeRO stage 1 already removes the full copy of the weights from each GPU
  • Stage 2 partitions the weights; stage 3 only adds offload
  • offload_param works with any ZeRO stage
  • train_batch_size is the batch each GPU processes
  • ZeRO-3 partitions state only among the GPUs inside one node
  • A higher ZeRO stage never adds communication to the step

Assumes DeepSpeed 0.19.7: overlap_comm's dynamic default, which stages accept each offload block and the 5e8 bucket defaults are checked against that release.

open as a page

What does PyTorch 2.14's FSDP2 fully_shard do to a model's parameters, and what changes for code written against FSDP1?

level: middleimportance: should knowfreq 15%

basics

~20 s

fully_shard swaps each parameter in place for a DTensor sharded on dim 0 across the mesh and all-gathers each call's group around forward and backward. FSDP1's wrapper, FlatParameter and ShardingStrategy give way to per-block calls and reshard_after_forward.

open as a page

With Megatron-Core 0.19, how would you set TP, PP, CP and DP for a 70B dense decoder on 32 nodes of eight 80 GB GPUs?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Tensor parallel 8 with sequence parallelism inside each node, pipeline parallel 4 across nodes with virtual stages, context parallel 1 at 8K tokens, leaving data parallel 256 / (8 x 4 x 1) = 8, plus the distributed optimizer.

open as a page

With expert parallelism in Megatron-Core or DeepSpeed, what all-to-all traffic does each MoE layer of a 64-expert model add, and where should the expert-parallel group sit?

level: seniorimportance: should knowfreq 6%

basics

~20 s

Each MoE layer adds a dispatch all-to-all sending tokens to the ranks holding their experts and a combine all-to-all bringing results back, mirrored in backward, every micro-batch. Keep the expert-parallel group inside one NVLink node.

open as a page

For a 70B decoder's 128K-token training phase, how do DeepSpeed-Ulysses and Megatron-Core context parallelism split attention, and what caps each one's degree?

level: seniorimportance: nice to knowfreq 3%

basics

~20 s

DeepSpeed-Ulysses all-to-alls Q, K and V from sequence shards to head shards, so key-value heads cap its degree. Megatron-Core context parallelism keeps sequence shards and passes KV chunks around a ring, capped by overlap and by sequence length divisibility.

open as a page

Before a 256-GPU pretraining cluster is built, how would you decide whether the program standardizes on FSDP2, DeepSpeed or Megatron-Core?

level: principalimportance: nice to knowfreq 4%

basics

~20 s

Decide on conditions, not brand: whether sharded data parallelism alone fits the model, whether the roadmap needs tensor, pipeline, context or expert parallelism, how the interconnect is shaped, and whether the team can rewrite model code into Megatron-Core's layers.

open as a page