A torch.compile'd 7B fine-tune in PyTorch 2.14 shows little speedup and slow early steps; how do you find graph breaks and recompiles, and decide if compiling pays?
answer
- one graph or many fragments
- guards fail, the function recompiles
- a limit, then no more variants
- explain() lists, fullgraph fails fast
- compile minutes against per-step seconds
basics
~10 sRun torch._dynamo.explain on the step or compile with fullgraph=True to find graph breaks (prints, .item(), data-dependent branches); stop shape-driven recompiles with bucketing or mark_dynamic; then weigh one-off compile time against per-step savings.
solid answer
~40 sA **graph break** ends the current graph where TorchDynamo cannot capture something, runs that piece in Python and starts a new graph — the step still works, but as small fragments with little to fuse. In a training step the usual culprits are `print`, a per-step `loss.item()`, branches on tensor values and calls into non-PyTorch C extensions. `torch._dynamo.explain(fn)(*inputs)` lists every break with its reason; `torch.compile(fn, fullgraph=True)` makes the first break an error. **Recompiles** happen when a guard fails — typically a new sequence length — and repeat up to `torch._dynamo.config.recompile_limit` times; bucket lengths or call `torch._dynamo.mark_dynamic` up front. Under DDP, breaks at bucket boundaries are DDPOptimizer's deliberate work. Then do the arithmetic: 6 minutes of compile saving 0.3 s per step pays back after 1,200 steps.
code
python · 18 linesimport torch
import torch._dynamo as dynamo
import torch.nn.functional as F
def forward_loss(model, inputs, labels):
logits = model(inputs)
loss = F.cross_entropy(logits.flatten(0, 1), labels.flatten())
print("loss", loss.item()) # graph breaks: .item() and print
return loss
# Lists every graph break with its reason (runs the function once)
explanation = dynamo.explain(forward_loss)(model, inputs, labels)
print(explanation)
# Once the breaks are fixed, keep it that way: the first break becomes an error
compiled_forward_loss = torch.compile(forward_loss, fullgraph=True)go deeper
Recall that torch.compile captures Python into graphs, that some code breaks a graph, and that the first steps include compile time.
Explain graph breaks and guards, name common breaking calls in a training step, and use explain and fullgraph=True to find them.
Diagnose a compiled fine-tune with no speedup: separate breaks from shape recompiles, fix them, and show the compile break-even arithmetic.
Decide where compilation belongs in a team's training stack, weighing short experiments and frequent restarts against long runs that repay it.
## What torch.compile does to a training step `torch.compile` hands a function or module to **TorchDynamo**, which reads its Python bytecode, captures the tensor operations into an **FX graph**, and passes the graph to a backend compiler that generates fused kernels. For training, AOTAutograd derives the matching backward graph, so compiling the model's forward also compiles its backward. Whether that pays off on a training step depends on two things: how much of the step lands in compiled graphs, and how often those graphs have to be rebuilt. ## Graph breaks When Dynamo meets something it cannot capture, it ends the current graph, runs that piece as ordinary Python, and starts a new graph after it — a **graph break**. The step still runs, but as several smaller compiled fragments with Python in between, and small fragments give the compiler little to fuse. Common causes in a training step: - `print(...)` or logging calls inside the compiled region. - Reading a GPU value on the host, such as `loss.item()`, which also forces a synchronization. - Control flow that depends on tensor values, such as `if x.sum() < 0:`. - Calls into C extensions other than PyTorch, which Dynamo cannot see into. How to find them: 1. `torch._dynamo.explain(fn)(*inputs)` runs Dynamo over the function and reports the graph count, the graph-break count, and the reason and user stack for every break. It executes the function, so run it on a scratch batch. 2. `torch.compile(fn, fullgraph=True)` turns the first graph break into an error — useful as a guard once a region is break-free. 3. `TORCH_LOGS="+dynamo"` prints every traced instruction for the hard cases. Not every break is a defect. When a model wrapped in DDP is compiled, **DDPOptimizer** deliberately splits the graph at the boundaries of DDP's gradient buckets, so the all-reduce of one bucket can overlap the backward of the next. `torch._dynamo.config.optimize_ddp=False` turns it off; the job still runs, but loses that overlap. ## Recompilations Dynamo protects each compiled graph with **guards** — checks on input shapes, dtypes and the Python values the trace depended on. When a guard fails on a new call, Dynamo compiles the function again, up to `torch._dynamo.config.recompile_limit` times; past that limit it stops producing new compiled variants of that function. A step that recompiles often shows slow, irregular early steps and a speedup that never arrives. The usual trigger in fine-tuning is shape. A code dataset with variable-length samples presents a new sequence length on many batches. Remedies: - **Bucket or pad** sequence lengths to a few fixed sizes, so each size compiles once. - **Mark the dimension dynamic** up front with `torch._dynamo.mark_dynamic(tensor, dim)`, so one graph serves every length instead of a static graph that later has to be replaced. - **Keep changing Python values out of the compiled function.** A step counter or flag the trace reads becomes something a guard checks. Raising `recompile_limit` only hides the symptom; past a point, recompiling costs more than it saves. ## Does compiling pay? Compilation is a one-off cost per process, and again for every new graph, paid during the first steps. Decide with measured numbers: 1. Compile overhead: the time of the first steps minus the steady-state step time. Suppose 6 minutes, or 360 s. 2. Steady-state step time, compiled against uncompiled: say 1.7 s against 2.0 s, a saving of 0.3 s per step. 3. Break-even: `360 ÷ 0.3 = 1,200` steps. A fine-tune of 20,000 steps recovers that many times over; a 500-step smoke test does not. Every restart pays again unless the compile caches survive it; `TORCHINDUCTOR_FORCE_DISABLE_CACHES=1` disables all compile caching, which is how you measure the true cold-start cost. When you profile a compiled job, set the profiler's `skip_first` past the compilation steps, or the trace will show the compiler rather than the step. ## Symptom table | Symptom | Likely cause | First tool | |---|---|---| | Little speedup, many short graphs | Graph breaks | `torch._dynamo.explain` | | Early steps slow, and slow again on new shapes | Recompiles from shape guards | Bucketing or `mark_dynamic` | | Speedup fades as new shapes keep arriving | Recompile limit reached | Find the failing guard | | Compiled DDP graph split into pieces | DDPOptimizer at bucket boundaries | Expected; `TORCH_LOGS='ddp_graphs'` to inspect |
- Your compiled DDP model shows several graphs per step although explain finds no breaks in your code. Is something wrong?Probably not. DDPOptimizer splits the compiled graph at DDP's bucket boundaries on purpose, so each bucket's all-reduce can start while the rest of backward runs. `TORCH_LOGS='ddp_graphs'` dumps those graphs. Turning it off with `torch._dynamo.config.optimize_ddp=False` still works but gives up the overlap.
- How do you measure the real cold-start compile cost of a job that restarts?Run with `TORCHINDUCTOR_FORCE_DISABLE_CACHES=1`, which disables every compile cache, and time the first steps against steady-state steps. Compare that with a normal restart where caches survive; the difference is what caching buys you on every restart.
- A graph break is an error, so the step cannot run with one.
- Every split graph in a compiled DDP model is a bug to remove.
- Raising recompile_limit fixes recompiles at no cost.
- The first steps' speed is the compiled speed.
- Logging loss.item() inside the compiled region is free.
Assumes PyTorch 2.14, where the recompilation cap is torch._dynamo.config.recompile_limit.