In PyTorch, what is the difference between an nn.Parameter and a registered buffer?

## The two kinds of state a module owns An `nn.Module` in PyTorch is a container for three things: submodules, parameters, and buffers. Submodules are other `nn.Module` instances. Parameters and buffers are both tensors, and the distinction between them is one of the most reliably asked PyTorch questions, because getting it wrong produces bugs that never raise an exception. A **parameter** is a tensor wrapped in `nn.Parameter`. Wrapping does two things: it sets `requires_grad=True` by default, and it makes `nn.Module.__setattr__` register the tensor in the module's internal `_parameters` dict when you assign it to an attribute. Registered parameters are what `model.parameters()` and `model.named_parameters()` yield — and that iterable is precisely what you pass to an optimizer, e.g. `torch.optim.Adam(model.parameters(), lr=1e-3)`. So "is it a parameter?" is operationally the same question as "will the optimizer update it?" A **buffer** is a plain tensor you register explicitly with `self.register_buffer("running_mean", torch.zeros(dim))`. It goes into the module's `_buffers` dict. It is yielded by `model.buffers()` / `model.named_buffers()`, never by `model.parameters()`, and therefore no optimizer ever touches it. If it changes, it changes because your own code assigned to it. ## What the two have in common This is the half candidates usually miss. Both parameters and buffers are: - **moved by `.to()`, `.cuda()`, `.float()`, `.half()`** — `nn.Module.to()` walks `_parameters` and `_buffers` recursively through all submodules and swaps the tensors in place. A tensor stored as a bare attribute is in neither dict, so it is silently left behind on the CPU. - **included in `state_dict()`** — checkpoints contain both, which is why loading a BatchNorm checkpoint restores its running statistics and not just its affine weights. - **visible to `model.apply(fn)` traversal indirectly**, in the sense that the module tree they hang off is walked. ## The failure mode ``` self.mask = torch.tril(torch.ones(n, n)) # bare attribute ``` This looks harmless. Then `model.cuda()` moves every parameter and buffer to the GPU but not `self.mask`, and the first forward pass raises `RuntimeError: Expected all tensors to be on the same device`. Worse, if you had assigned a bare tensor that you *intended* to train, there is no error at all — it simply never appears in `model.parameters()`, the optimizer never updates it, and the model trains around a frozen tensor. That silent version is the one interviewers like, because it costs a real training run to discover. The symmetric mistake is wrapping something in `nn.Parameter` that should not be learned — a positional-encoding table or a normalisation constant. Now the optimizer drifts it, and results change subtly between runs. ## persistent=False `register_buffer(name, tensor, persistent=False)` registers the buffer for device movement but excludes it from `state_dict()`. Use it for anything that is a pure function of the config and can be rebuilt at construction time: causal attention masks, rotary embedding tables, precomputed windows. It keeps checkpoints smaller and, more importantly, keeps them loadable when you later change the sequence length the mask was sized for. ## Freezing versus buffering A related question: if you want a tensor that is trainable-shaped but currently frozen, do you make it a buffer? No. Keep it an `nn.Parameter` and set `param.requires_grad = False`, or simply exclude it from the parameter group you pass to the optimizer. It stays a parameter — same `state_dict` key, same semantics — and you can unfreeze it later. Turning it into a buffer changes the checkpoint key and the meaning of the attribute. ## What to check when debugging Three one-liners settle almost every question in this area: - `[n for n, _ in model.named_parameters()]` — is the tensor going to be optimized? - `[n for n, _ in model.named_buffers()]` — is it going to move with the module? - `list(model.state_dict().keys())` — is it going to survive a save/load round trip? If a tensor appears in none of the three, it is a bare attribute, and both device movement and checkpointing will skip it.