skip to content

With torch.profiler in PyTorch 2.14, how do you capture a few representative training steps, and what do you look for in the trace?

level: middleimportance: must knowfreq 18%

answer

  1. steps, not seconds
  2. skip the compile and warm-up steps
  3. warmup traced then thrown away
  4. prof.step() ends every iteration
  5. gaps on the GPU stream

basics

~10 s

Wrap the loop in torch.profiler.profile with schedule(skip_first, wait, warmup, active, repeat) and call prof.step() each iteration; then read the exported trace for gaps on the GPU stream and what the CPU did meanwhile.

solid answer

~40 s

Pass `schedule=torch.profiler.schedule(skip_first=..., wait=..., warmup=..., active=..., repeat=...)` and `on_trace_ready=tensorboard_trace_handler(dir)` to `torch.profiler.profile`, and call `prof.step()` at the end of every iteration — the schedule counts steps, not time. `skip_first` jumps past compilation and allocator growth; each `warmup` step is traced and discarded because the first traced steps are skewed; `active` steps are recorded; `repeat=0` (the default) keeps cycling. Without a schedule, PyTorch 2.14 records every step. In the trace, look for idle stretches on the GPU stream and what the CPU thread was doing then: a `cudaStreamSynchronize` from a per-step `loss.item()`, long runs of tiny kernels (launch-bound), communication that does not overlap compute, or waits for input. Keep `record_shapes`, `profile_memory` and `with_stack` to short windows; they add overhead.

code

python · 18 lines
python
from torch.profiler import ProfilerActivity, profile, schedule, tensorboard_trace_handler

sched = schedule(skip_first=10, wait=5, warmup=1, active=3, repeat=2)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=sched,
    on_trace_ready=tensorboard_trace_handler("./traces/rank0"),
    record_shapes=True,
    with_stack=True,
) as prof:
    for step, batch in enumerate(loader):
        train_step(model, batch, optimizer)
        prof.step()  # one call per iteration: the schedule counts these
        if step >= 27:
            break

print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=10))

go deeper

for a junior

Recall that the profiler records CPU ops and GPU kernels per step, and that prof.step() must be called once per iteration.

for a middle

Explain each schedule phase — skip_first, wait, warmup, active, repeat — work out which steps get recorded, and say why warmup output is discarded.

for a senior

Read a trace for idle GPU gaps and name the host-side cause: synchronizing reads, launch-bound tiny kernels, exposed communication, input waits.

for a principal

Decide what profiling runs routinely on long jobs — a cheap repeating schedule on one rank — so a throughput regression has evidence when it appears.

## Why the profiler needs a schedule `torch.profiler.profile` records operator calls on the CPU and kernels on the GPU. Without a `schedule`, PyTorch 2.14 records every step from the moment the context opens. For a training job that is the wrong window: the first steps carry one-off costs — `torch.compile` building graphs, the caching allocator growing its pool, libraries choosing algorithms — and hundreds of recorded steps produce a trace too large to open. A schedule records a few steady-state steps and skips the rest. The profiler counts **steps**, not seconds. A step ends each time the loop calls `prof.step()`, so that call belongs at the end of every iteration. ## The schedule's phases `torch.profiler.schedule(*, wait, warmup, active, repeat=0, skip_first=0, skip_first_wait=0)` returns a function that maps a step number to an action. | Parameter | What happens in those steps | |---|---| | `skip_first` | Nothing; skipped once at the start (default 0) | | `wait` | Profiler idle at the start of each cycle | | `warmup` | Profiler traces, but the results are discarded, because the first traced steps carry extra overhead | | `active` | Events are recorded; at the end of the cycle `on_trace_ready` is called | | `repeat` | Number of cycles; 0, the default, cycles for as long as the job runs | | `skip_first_wait` | When non-zero, the first cycle skips its `wait` phase | `warmup=0` is accepted but warns that results can be skewed, and `active` must be at least 1. Worked through for `schedule(skip_first=10, wait=5, warmup=1, active=3, repeat=2)`, with steps numbered from zero: 1. Steps 0–9 are skipped. 2. Steps 10–14 wait, step 15 warms up, steps 16–18 are recorded and handed to `on_trace_ready`. 3. Steps 19–23 wait, step 24 warms up, steps 25–27 are recorded and handed over. 4. With `repeat=2`, nothing after step 27 is recorded. ## Saving and reading the trace `on_trace_ready` receives the profiler at the end of each cycle. `tensorboard_trace_handler(dir_name)` writes one `*.pt.trace.json` file per cycle, prefixed with `worker_name` — `hostname_pid` by default, which keeps processes apart — into a directory TensorBoard can read. `prof.export_chrome_trace(path)` writes a single Chrome-trace file by hand, and `prof.key_averages().table(sort_by="self_cuda_time_total")` gives the same data as a table. In the timeline, each step appears as a `ProfilerStep#N` range. What to look for: - **Gaps on the GPU stream.** Idle stretches between kernels mean the GPU waited on the host. Check what the CPU thread was doing at that moment. - **A CPU thread blocked in `cudaStreamSynchronize`.** Something read a GPU value on the host — often `loss.item()` for logging every step — and the CPU stopped queueing work until the GPU drained. - **Kernels shorter than their launch overhead.** Long runs of tiny elementwise kernels, or an optimizer step made of thousands of small kernels, mean the step is launch-bound; fusion is the remedy. - **Communication kernels that do not overlap compute.** Exposed collectives add step time; tuning them is the collectives layer's job, but the trace is where you see them. - **Input waits.** Time spent in the data-loading iterator before the forward starts points to the input pipeline. ## The switches that add overhead - `record_shapes=True` records operator input shapes; it holds tensor references temporarily and can add copies. - `profile_memory=True` tracks tensor allocations and frees. - `with_stack=True` records file and line for each op, and in 2.14 also records `nn.Module` events for eager models. - `with_flops=True` estimates FLOPs for matrix multiplications and 2D convolutions. - `with_modules` is deprecated in 2.14: it only collects data for TorchScript models and does nothing in eager mode. `export_memory_timeline` is deprecated too. Turn the expensive switches on for a short `active` window, not for the whole run. ## On a multi-GPU node Every rank runs its own profiler and writes its own trace. Profiling one rank, or two, is usually enough to find a stall that affects them all, and keeps the files manageable. Give each rank its own directory or `worker_name` so traces do not collide. When the question is the whole process rather than PyTorch's operators — CPU threads, the CUDA API, OS runtime calls — a system-wide timeline from Nsight Systems complements the profiler.

  • When would you reach for Nsight Systems instead of torch.profiler?
    When you need the whole process on one timeline — CPU threads, CUDA API calls, OS runtime, NVTX ranges, other libraries — rather than PyTorch's operators. `nsys profile -t cuda,nvtx --pytorch=autograd-nvtx -o step_trace python train.py` annotates PyTorch's autograd ops with NVTX ranges; `--capture-range=cudaProfilerApi` limits collection to the region the script brackets with the CUDA profiler start and stop calls.
  • Why not simply profile the first ten steps of the run?
    The first steps are not the steady state. A compiled model spends them building graphs, the caching allocator is still growing its pool, and libraries are still choosing algorithms. A trace of those steps measures setup, not training; `skip_first` or a long enough `wait` moves the recorded window past them.
  • The trace shows a GPU gap before every optimizer step while the CPU sits in a synchronize call. What is the likely fix?
    Something on the host reads a GPU value every step — typically `loss.item()` or a printed metric. Each read waits for the GPU to finish and stops the CPU from queueing the next kernels. Log every N steps, or keep the running value on the device and read it rarely.

  • Profile from step zero; the first steps are representative.
  • Events from the warmup phase are kept in the saved trace.
  • The schedule counts seconds rather than calls to prof.step().
  • record_shapes, profile_memory and with_stack cost nothing, so leave them on.
  • with_modules gives the module hierarchy for eager models in PyTorch 2.14.
  • A GPU stream with no gaps proves the step is efficient.

Assumes PyTorch 2.14: schedule() takes wait, warmup, active, repeat, skip_first and skip_first_wait, and with_modules and export_memory_timeline are deprecated.