skip to content

A dense model moving from 4K- to 64K-token documents and a mixture-of-experts model are both tensor- and pipeline-split yet run out of memory — what do you split next for each, and what does it cost?

level: seniorimportance: nice to knowfreq 24%

answer

  1. which memory term grew
  2. sequence axis versus expert axis
  3. keys and values must travel
  4. dispatch and combine, forward and backward
  5. tokens x hidden x experts per token

basics

~20 s

Split by what grew. Long-sequence activations call for a context split: each device holds a slice of the tokens, and attention exchanges keys and values. Expert weights call for an expert split: whole experts per device, paid in per-layer all-to-alls.

solid answer

~50 s

First ask which term grew. For the dense model, weights are unchanged and activations rose about 16x, so I add a **context split**: each device keeps `1/N` of every sequence's tokens at every layer — 64K over 8 devices is 8K tokens each. Weights stay replicated, their gradients summed across the group. Attention needs every token's keys and values, so they are passed around a ring or regrouped by heads; that hides behind compute only when chunks are long. For the mixture-of-experts model, expert weights are the wall, so I add an **expert split**: whole experts on different devices. Tokens then travel to their experts and back — a dispatch and a combine all-to-all per expert layer, again in the backward pass, every micro-batch. Its volume tracks tokens x hidden x experts per token, and it degrades sharply once the group leaves the fast link inside one machine.

go deeper

for a junior

Recall that there are more ways to split training than data, layers and matmuls: a model can also be split by sequence and, for mixture-of-experts layers, by expert.

for a middle

Explain what each split cuts and what it adds: a context split cuts per-token activations but forces keys and values to move inside attention; an expert split cuts expert weights but sends tokens through a dispatch and a combine all-to-all.

for a senior

Diagnose before splitting: name which memory term grew, choose the axis that cuts it, and cost the traffic, including the backward pass, per micro-batch, and whether the expert group stays on the fast link inside one machine.

for a principal

Frame it as trading memory for communication on a fixed interconnect: when the expert group would spill across machines, weigh replicating the expert set per machine against the all-to-all's cost on the slower network.

## Start from what grew Two programs share a pool of eight-device machines. Both models are already **tensor-split** (each matmul cut across devices on the fast link inside a machine) and **pipeline-split** (consecutive layers on different stages), and both still run out of memory. First ask *which term in the memory budget grew*, because each remaining split cuts a different one: - The **dense model** is moving from 4K-token to 64K-token documents. Its weights, gradients and optimizer state did not change. What grew is **activations**: most of what a layer stores for the backward pass scales with the number of tokens, so 16 times longer sequences mean roughly 16 times the activation memory. - The **sparse mixture-of-experts model** has expert weights that make up most of its parameters. Its wall is **expert weights and their optimizer state**. ## The context split: cut the sequence A **context split** (also called a sequence split) of degree `N` gives every device in the group `1/N` of each sequence's tokens, at every layer. The arithmetic is direct: a 64K-token sequence over a context degree of 8 leaves each device 8K tokens — twice its old 4K load instead of sixteen times. Weights are *not* cut: every device in the group holds the same weights, so their gradients are summed across it, as data-parallel replicas do. Most layers do not care that the tokens are split — a feed-forward block or a normalization works on each token independently. **Attention** is the exception: each query must see the keys and values of every token it may attend to, and most of those now live on other devices. Two ways to pay for that: 1. **Pass key and value chunks around a ring of devices**, computing attention against each chunk as it arrives and overlapping the next transfer with the current compute. 2. **Regroup by heads**: an all-to-all turns "my tokens, all heads" into "all tokens, my heads", each device runs full attention for its heads, and a second all-to-all turns it back. The backward pass repeats the exchange for the key and value gradients. The split **pays only while each chunk carries enough compute to hide the exchange**: attention compute between two chunks grows with the product of their lengths, the data moved with one chunk's length, so the compute-to-traffic ratio improves in proportion to chunk length. At 64K tokens over 8 devices a chunk is 8K tokens; 4K-token documents at the same degree would leave 512-token chunks, sixteen times worse, with the exchange exposed on the critical path. ## The expert split: place whole experts An expert layer holds many independent feed-forward networks, and each token uses only a few of them. Slicing *every* expert thin with a tensor split divides the weights but turns already small expert matmuls into tiny ones and adds a collective to every expert layer. The **expert split** of degree `N` instead places **whole experts** on different devices — with 64 experts over 8 devices, each holds 8. The group's devices usually hold *different* data, so attention layers run as ordinary replicas while each expert lives on one device. If experts are, say, 90% of the parameters, an expert degree of 8 leaves each device `0.10 + 0.90 / 8 ≈ 0.21` of its former parameter memory, optimizer state shrinking in step. The price is that **tokens must travel to their experts and back**. Every expert layer runs a **dispatch all-to-all** (send each token's hidden vector to the devices holding its chosen experts) and a **combine all-to-all** (bring the expert outputs home), and the backward pass runs both again for the gradients — at least four all-to-alls per expert layer, every micro-batch. ## What the all-to-all costs | | context split | expert split | |---|---|---| | cuts | activations, by sequence length | expert weights and their state | | leaves whole | weights | each expert's matmuls | | adds | key/value exchange inside attention | dispatch and combine all-to-all per expert layer | | pays when | chunks are long enough to hide the exchange | the group stays on the fast link | The all-to-all's **volume tracks tokens × hidden size × experts chosen per token**, not the expert count. Doubling the experts from 64 to 128 with two chosen per token moves the same bytes to more destinations; choosing four instead of two doubles it. What makes it expensive is its **shape**. It is pairwise traffic: every device sends a different piece to every other, and the step ends only when the **slowest link** and the **busiest device** finish; uneven routing makes the busiest device the straggler. The expert computation cannot start until its tokens arrive, so the dispatch is hard to hide. That is why the expert split **stops paying once the group spills off the fast link inside a machine**. Widen a group from 8 devices in one machine to 16 across two, and under even routing about half of each device's token copies cross the slower network between machines — shared by a machine's eight devices — at least four times per expert layer per micro-batch. ## Common wrong moves - **More micro-batches**: once a micro-batch is one sequence, more of them shrink the pipeline bubble, not a sequence's activations. - **A wider tensor degree across machines**: it mainly thins weights that did not grow, and puts a blocking collective in every block onto the slower network. - **Offloading optimizer state**: it frees a fixed amount that does not grow with sequence length. - **Recomputation alone**: it helps, but one 64K-token layer's working set is still sixteen times the 4K one.

  • What happens to the expert split's cost when its group is widened from one eight-device machine to sixteen devices across two?
    The volume per device barely changes, but where it goes does: under even routing about half of each device's dispatched token copies now cross the slower network between machines, shared by all eight devices there. Each all-to-all ends only when its slowest link and busiest device finish, at least four times per expert layer per micro-batch, so throughput drops sharply. Keeping the group inside one machine, with a copy of the expert set per machine, trades memory for speed.
  • Why would the same context split that pays at 64K tokens hurt a 4K-token run?
    Attention compute between two chunks grows with the product of their lengths, while the keys and values moved grow with one chunk's length, so long chunks hide the exchange. At degree 8, 64K tokens give 8K-token chunks; 4K tokens give 512-token chunks, a sixteen-times worse compute-to-traffic ratio. The exchange sits exposed on the critical path, and the activations it would cut were never the wall.
  • Does a context split change how weight gradients are communicated?
    Yes. Every device in the context group holds the same weights and computes gradients from a different slice of tokens, so those gradients must be summed across the group, exactly as data-parallel replicas sum theirs. That reduction is in addition to the key and value exchange inside attention, which the backward pass also repeats for their gradients.

  • A context split divides the weights too, so it fixes any memory wall.
  • More micro-batches fix memory once one 64K-token sequence no longer fits.
  • Expert all-to-all volume grows with the number of experts in the layer.
  • An expert split moves expert weights to wherever the tokens are.
  • Spanning machines with the expert group is free because the all-to-all hides behind compute.
  • A context split pays off at any sequence length.