skip to content

How do you capture a CUDA memory snapshot of a PyTorch 2.14 fine-tune that OOMs hours in, and tell fragmentation from a leak in it?

level: seniorimportance: should knowfreq 12%

answer

  1. record before the first step
  2. cap the event history
  3. dump when the OOM is raised
  4. rising floor versus free slivers
  5. growable segments in the allocator config

basics

~10 s

Call torch.cuda.memory._record_memory_history(max_entries=...) at launch, catch the OOM and call _dump_snapshot, then open it in the memory viewer: a rising per-step floor is a leak; segments full of unusable free slivers are fragmentation.

solid answer

~40 s

Start `torch.cuda.memory._record_memory_history(max_entries=...)` before training — `max_entries` defaults to `sys.maxsize`, so cap it and the history becomes a ring buffer of the latest events. Catch `torch.OutOfMemoryError`, call `torch.cuda.memory._dump_snapshot(path)`, re-raise, and drop the file into the viewer at pytorch.org/memory_viz. In the Active Memory Timeline a leak is a per-step sawtooth whose low point climbs; hovering a block shows the stack that allocated it. In the Allocator State History at the OOM event, fragmentation is segments with plenty of free space split into pieces too small for the request — `memory_stats()` shows it as large `inactive_split_bytes`. For fragmentation from shifting tensor sizes, set `PYTORCH_ALLOC_CONF=expandable_segments:True` (experimental, default `False`; `PYTORCH_CUDA_ALLOC_CONF` is only a backward-compatible alias in 2.14).

code

python · 12 lines
python
import os
import torch

# Keep only the most recent alloc/free events, as a ring buffer
torch.cuda.memory._record_memory_history(max_entries=200_000)

try:
    for step, batch in enumerate(loader):
        train_step(model, batch, optimizer)
except torch.OutOfMemoryError:
    torch.cuda.memory._dump_snapshot(f"oom_{os.getpid()}.pickle")
    raise

go deeper

for a junior

Recall the two calls — _record_memory_history to start recording, _dump_snapshot to write the file — and where the file is opened.

for a middle

Explain segments, blocks and caching, why reserved exceeds allocated, and what expandable_segments changes for allocations whose sizes shift between steps.

for a senior

Capture an OOM hours into a run with a capped ring buffer, then show in the timeline and allocator history whether it is a leak or fragmentation.

for a principal

Decide whether bounded memory-history recording ships on every long training job by default, weighing its small cost against an unrepeatable failure.

## What a memory snapshot records PyTorch's CUDA **caching allocator** requests large **segments** of memory from the driver and splits them into **blocks** for individual tensors. A freed block stays cached in its segment for reuse instead of going back to the driver, which is why `nvidia-smi` shows more memory in use than live tensors occupy. A **memory snapshot** captures that state — every segment, every block, whether it is allocated or free, and the stack trace that allocated it — plus, when history recording is on, a timeline of the allocation and free events that led up to the dump. Two calls in `torch.cuda.memory` do the work: - `_record_memory_history(...)` starts recording stack traces and the event history. `enabled`, `context` and `stacks` all default to `"all"`, which keeps the history of every alloc and free and both Python and C++ frames. - `_dump_snapshot(filename)` pickles the current snapshot to a file. The viewer at pytorch.org/memory_viz opens it locally in the browser, without uploading it. ## Capturing an OOM that strikes hours in 1. **Start recording at launch**, before the first step, so the events leading to the failure exist when it happens. 2. **Cap the history.** `max_entries` defaults to `sys.maxsize`, so on a multi-hour run the history — several KB per entry — grows without bound. With a cap the buffer becomes a ring that keeps only the most recent events, which are the ones that explain the OOM. Snapshot files scale with `max_entries` and stack depth and can reach gigabytes. 3. **Dump on failure.** Catch `torch.OutOfMemoryError` around the training loop, call `_dump_snapshot`, then re-raise so the job still fails visibly. 4. **One file per process.** On a multi-GPU node each rank sees only its own device, so each writes its own file. Python trace collection costs about 2 µs per trace, cheap enough that leaving history on for a long production job you may need to debug is reasonable. ## Reading it: leak or fragmentation | Evidence | Points to a leak | Points to fragmentation | |---|---|---| | Active Memory Timeline | A per-step sawtooth whose low point climbs step after step | A sawtooth that returns to the same floor each step | | Allocator State History at the OOM event | Segments packed with live blocks | Segments with plenty of free space, split into slivers smaller than the failed request | | `memory_stats()` | `allocated_bytes.all.current` keeps rising between steps | Large `inactive_split_bytes.all.current` ("Non-releasable memory" in `memory_summary()`) | | Stack traces on hover | The same allocation site survives step after step | Many slightly different sizes from the same layers | Hovering a block shows its allocation stack trace, so a leak usually leads straight to the line that holds references — a list of losses kept together with their autograd graphs, for example. Fragmentation is common when tensor sizes shift between steps. A code fine-tune that packs variable-length samples allocates slightly different activation sizes each step. A larger request may not fit the free tail of an existing segment, so the allocator asks for a new segment; the leftover tails pile up as slivers in every layer, and eventually a request fails although the free total would cover it. ## The allocator settings The caching allocator reads `PYTORCH_ALLOC_CONF`, a comma-separated list of `option:value` pairs. In PyTorch 2.14, `PYTORCH_CUDA_ALLOC_CONF` is its alias, provided only for backward compatibility. | Option | What it does | Notes | |---|---|---| | `expandable_segments:True` | Keeps one segment per stream that grows as needed, so differently sized allocations tile into it | Experimental; default `False` | | `max_split_size_mb:<n>` | Stops the allocator splitting blocks larger than `n` MB | A last resort for jobs showing many inactive split blocks | | `garbage_collection_threshold:<fraction>` | Starts reclaiming cached free blocks once usage passes that fraction | Default 1.0 | | `backend:cudaMallocAsync` | Switches to CUDA's stream-ordered allocator | `max_split_size_mb` and `garbage_collection_threshold` are then ignored | For a fragmentation OOM, `expandable_segments:True` in the launch environment is the first thing to try: re-run, re-snapshot and confirm the free slivers are gone. A leak needs a code fix; no allocator option cures it. ## What the snapshot cannot see The snapshot covers only memory that went through PyTorch's allocator. NCCL's communication buffers, libraries that call `cudaMalloc` directly and the CUDA context itself do not appear. If the device is full while reserved memory sits well below its total, compare `torch.cuda.device_memory_used()` with `torch.cuda.memory_reserved()` to size what lives outside the allocator.

  • Why cap max_entries instead of keeping the default?
    The default is `sys.maxsize`, so on a job that runs for hours the history grows without bound at several KB per entry, costing host memory and producing snapshot files in the gigabytes. A cap turns it into a ring buffer that keeps the latest events — the ones that lead to the OOM — at a fixed cost.
  • The device is full, but the snapshot's reserved memory is well below the GPU's capacity. Where is the rest?
    Outside PyTorch's caching allocator: NCCL's communication buffers, libraries that call `cudaMalloc` directly, and the CUDA context. The snapshot cannot show them. Compare `torch.cuda.device_memory_used()` with `torch.cuda.memory_reserved()` to see how much lives there.
  • When would you set max_split_size_mb instead of expandable_segments?
    As a last resort, when a job keeps dying with a large amount of inactive split memory and the native allocator is in use. It stops blocks larger than the limit from being split, which can reduce fragmentation at a performance cost that ranges from none to substantial. Under `backend:cudaMallocAsync` it is ignored.

A car park with enough empty spaces in total for a coach, but every space is a gap between parked cars; expandable segments is a car park that lengthens its last row, so the coach always has room at the end.

  • PYTORCH_CUDA_ALLOC_CONF is the primary allocator variable in PyTorch 2.14.
  • expandable_segments is already on by default.
  • Reserved memory far above allocated memory always means a leak.
  • Calling torch.cuda.empty_cache() frees memory held by live tensors.
  • The default history size is small enough to leave on for hours.
  • The snapshot also shows NCCL's buffers next to PyTorch's tensors.

Assumes PyTorch 2.14: PYTORCH_ALLOC_CONF is primary with PYTORCH_CUDA_ALLOC_CONF as an alias, and expandable_segments is still experimental and off by default.