Why does PyTorch's backward() accumulate into .grad instead of replacing it?

## The rule `backward()` does not assign to `.grad`; it does `+=`. If `.grad` is `None`, autograd allocates a tensor and stores the result. If a tensor is already there, the new gradient is added element-wise on top. ``` w = torch.tensor([1.0], requires_grad=True) (w * 2).sum().backward() w.grad # tensor([2.]) (w * 3).sum().backward() w.grad # tensor([5.]) -- 2 + 3, not 3 ``` ## Why summing is the right default This is not an arbitrary API choice, it falls out of the mathematics. If a variable feeds several downstream expressions, the multivariable chain rule says its derivative is the sum over all paths. Consider a weight-tied model where the same embedding matrix is used by the input layer and the output projection: gradient arrives at that matrix along two distinct routes and the correct update uses both. Inside a single graph, autograd's accumulation node for that leaf simply adds each incoming contribution as it arrives. Making the cross-call behaviour identical means there is no seam between "one backward with two paths" and "two backwards with one path each". That uniformity buys several patterns for free: - **Multiple loss terms computed separately.** `loss_a.backward(retain_graph=True)` then `loss_b.backward()` gives the same parameter gradients as `(loss_a + loss_b).backward()`, which is useful when the two terms are produced by different parts of a system. - **Splitting a batch that will not fit in memory.** Run several smaller forward/backward passes before updating; the summed gradient corresponds to the larger batch. (Whether to scale the loss by 1/N depends on whether your loss already averages over the batch.) - **Multi-task heads** whose losses are computed and back-propagated at different points in the step. ## The cost: persistent mutable state Because `.grad` survives across iterations, it behaves like an accumulator that you own. Forget to clear it and step N's update is computed from the sum of steps 1..N. Nothing raises. The loss curve is merely worse than it should be — often just "unstable, needs a lower learning rate", which is why this bug can live in a codebase for months. Two ways to clear: - `p.grad = None` — drops the reference. The next backward allocates fresh. This is cheaper (no elementwise kernel), lets the memory be reclaimed, and means untouched parameters read as `None` rather than as stale zeros. - `p.grad.zero_()` — writes zeros in place, keeping the buffer allocated. Necessary only if something else holds a reference to that exact tensor. In current PyTorch, `zero_grad()` defaults to the `None` form (`set_to_none=True`). One consequence catches people out: any code that assumes `.grad` is a tensor must handle `None`, and optimizers with momentum treat a `None` gradient as "no update for this parameter this step" rather than as an update with a zero gradient — which for some optimizers is a meaningfully different thing, since a zero gradient still decays momentum and can still apply weight decay. ## Related autograd behaviours worth separating Accumulation is often confused with two other things: - **`retain_graph=True`** controls whether the *graph* survives a backward pass, not whether gradients accumulate. Gradients accumulate whether or not you retain the graph; you need `retain_graph` only when you will call backward through the *same* graph again. - **`create_graph=True`** makes the backward pass itself differentiable, so you can take gradients of gradients. It also means the resulting `.grad` tensors carry their own graph, which holds memory — a common leak source in meta-learning code. And if you want a gradient *without* touching `.grad` at all, `torch.autograd.grad(outputs, inputs)` returns the values directly and accumulates nothing. That is the cleaner primitive for computing a gradient penalty or an influence score inside a step that also does ordinary training, precisely because it does not contaminate the accumulator the optimizer will read.