What do model.train() and model.eval() change in a PyTorch model?

## A flag, not a mode switch `model.train()` and `model.eval()` are thin: they set the boolean attribute `training` on the module and recurse into every registered submodule. `model.eval()` is exactly `model.train(False)`. Nothing is compiled, nothing is frozen, no graph is rebuilt. The visible behaviour change comes entirely from layers that branch on `self.training` inside their own `forward()`. That framing matters because it tells you the failure mode: any layer you write yourself that should behave differently at inference has to read the flag itself, and any third-party layer that ignores it will not change behaviour no matter which call you make. ## Dropout `nn.Dropout(p)` in train mode zeroes each activation independently with probability `p` and divides the survivors by `1 - p` ("inverted dropout"), so the expected value of the output matches the input. In eval mode it is the identity function — no zeroing, no rescaling. This is why evaluating a model that was left in train mode produces a validation loss that is both noisy and worse than the truth: you are measuring a randomly crippled network. It is also why the same input produces a different output on every call in train mode, which people often mistake for nondeterminism in the data pipeline. ## BatchNorm `nn.BatchNorm1d/2d/3d` are the more dangerous half. In train mode the layer normalizes the batch using the mean and variance **of that batch**, and updates its buffers `running_mean` and `running_var` with an exponential moving average controlled by the `momentum` argument. In eval mode it normalizes using those buffers and updates nothing. Two consequences follow. First, in train mode a sample's output depends on the other samples in its batch — feed a batch of size 1 to a `BatchNorm` layer in train mode and the variance is zero or undefined, which is why tiny batches destabilize BatchNorm models. Second, if you run validation without calling `eval()`, the running statistics are updated with validation data. That is a subtle form of contamination that survives into the next training epoch and into the saved checkpoint, because those buffers are part of `state_dict()`. `nn.LayerNorm`, by contrast, normalizes each sample over its own feature dimensions and keeps no running statistics, so it is genuinely mode-independent — one reason transformer stacks are less sensitive to this bug than convolutional ones. ## What the calls do not do The single most common misconception is that `eval()` turns off gradient tracking. It does not. A forward pass in eval mode still builds the autograd graph, still allocates the intermediate tensors needed for a backward pass, and still costs the memory. To actually skip that you need a context manager: `torch.no_grad()`, or `torch.inference_mode()` which is slightly stricter and slightly faster. Validation code therefore needs both: - `model.eval()` for correct layer behaviour, and - `with torch.no_grad():` for memory and speed. Equally, `eval()` does not freeze parameters. `requires_grad` is untouched, and if you call `backward()` and `step()` while in eval mode the weights update perfectly happily. ## Symptoms of getting it wrong - Validation loss noticeably worse than training loss with no other sign of overfitting, and different on every run → `eval()` was never called. - Inference results that shift depending on how requests are batched → the model is serving in train mode with BatchNorm. - Validation slowly degrading over epochs while training improves → running statistics being polluted by validation batches. - Out-of-memory during validation on batches the training loop handles fine → missing `no_grad`, because activations are being retained for a backward pass that never comes. ## Fine-tuning nuance When fine-tuning a pretrained backbone on a small dataset, teams often want BatchNorm to keep its pretrained statistics rather than re-estimate them from small, unrepresentative batches. The idiom is to call `model.train()` for the whole model and then call `.eval()` on the BatchNorm modules specifically — usually by iterating modules and checking `isinstance(m, nn.modules.batchnorm._BatchNorm)`. Note that this only stops the statistics updating; the affine `weight` and `bias` still train unless you also set `requires_grad = False` on them. Getting this distinction right — buffers versus parameters — is a strong senior signal. ## The checkpoint angle Because `running_mean` and `running_var` are buffers rather than parameters, they are saved by `state_dict()` but never seen by the optimizer. A model whose running statistics were corrupted therefore carries that corruption in its checkpoint, and no amount of optimizer-state restoration fixes it.