Regularization and Normalization

Random label-preserving transforms show the network a different version of each image every epoch, so it cannot memorise exact pixels. This effectively enlarges the training set and bakes in invariances such as small shifts and rotations, cutting variance.

Dropout zeroes each hidden unit at random on every training step, so no unit can rely on a particular other unit being present. Training therefore fits many thinned, weight-sharing subnetworks whose behaviour is averaged at evaluation.

LayerNorm computes a mean and variance across the feature dimensions of each sample on its own, then rescales with a learned per-feature gain and bias. No other sample enters the statistic, so batch size and batch composition cannot change the output.

Weight decay pulls every decayed weight slightly toward zero on each update, on top of the gradient step. The result is a smaller-norm solution: the network fits with less extreme weights, which usually improves held-out performance.

One mean and one variance per channel, pooled over every element of that channel in the mini-batch: all samples and all spatial positions. A layer after a 64-channel convolution holds 64 means, 64 variances, 64 scales and 64 shifts.

It divides every surviving activation by the keep probability, one minus the drop rate. That makes the layer's expected output equal its unmasked value, so evaluation runs the plain forward pass with no mask and no rescaling.

Patience keeps the run going for several epochs after its best validation score, so the final weights are exactly the ones that already got worse. Restore the best epoch's weights, saved together with the normalization layers' running statistics.

Label smoothing replaces the one-hot target with a blend of one-hot and uniform: the true class is asked for about 1 minus epsilon, every other class for epsilon divided by the number of classes.

Post-norm normalizes after the residual addition: out = Norm(x + F(x)). Pre-norm normalizes the branch input instead: out = x + F(Norm(x)). Pre-norm leaves an un-normalized identity path running the whole depth of the stack, which is why deep stacks train far more easily.

mixup draws a mixing weight lam from a Beta distribution and emits one blended example: pixels lam*x_i + (1-lam)x_j, with the target lamy_i + (1-lam)*y_j. The label is interpolated by the same weight, never rounded back to a single class.

Because shrinking them regularizes almost nothing and can actively hurt. Biases and normalization scale and shift are few in number, do not control how sharply the function bends, and pulling a normalization scale toward zero throttles that layer's output.

With two samples per batch the mean and variance are almost pure noise, and updating on them overwrites far better statistics inherited from large-batch pretraining. Freezing keeps the inherited values and makes each layer a stable fixed affine transform.

Ask whether an expert labeller would give the transformed image the same label. Brightness and contrast jitter mimic exposure differences and are safe on chest radiographs; a horizontal flip is not, since mirroring destroys laterality.

Inverted dropout inflates the variance of the activations it emits, but only during training. Batch norm's running variance is therefore estimated on inflated inputs and is too large for the mask-free activations at evaluation, so outputs come out shrunk.

It usually reduces miscalibration, because it removes the incentive to drive confidence toward 1. But it is a blunt global cap, not a calibration procedure: too much smoothing turns overconfidence into underconfidence, and it never guarantees confidence tracks accuracy.

The group count is a dial between whole-sample statistics at one group and per-channel statistics at one group per channel. Pick a middle value, or a fixed channels-per-group, and decide by what the statistic destroys rather than by estimator noise.

It grows. Each pre-norm block adds a branch output whose size is set by its own normalized input, not by the stream, so contributions accumulate and the stream's standard deviation rises roughly with the square root of depth. That is why a final normalization must sit before the output head.

Read the held-out curve, not the training curve. Across a logarithmic sweep the held-out metric is U-shaped: too little decay changes nothing, too much underfits. Pick from the flat top of that curve, then re-sweep when data volume or schedule changes.

Monitor the quantity the model is judged on, not the surrogate loss, whenever it can be computed every epoch. Track its own direction of improvement and its own noise scale, and keep the checkpoint that is best on it.

RMSNorm drops the mean subtraction and usually the bias, dividing each feature by the vector's root mean square and applying a learned gain. It assumes re-scaling invariance is what makes normalization work, and that re-centring can be given up.

The original internal-covariate-shift story did not survive testing. Later work showed the layer mainly smooths the loss landscape, so gradients stay predictive and larger learning rates remain stable, with the batch-sampled noise adding a mild regularizing effect.

Only when the accuracy gain is worth ten forward passes per input. Averaging predictions over ten label-preserving views cuts prediction variance for a fixed model, which suits offline or batch scoring far better than a latency-bound online request path.

Neighbouring positions in a feature map are strongly correlated because their receptive fields overlap, so zeroing scattered individual activations leaves the same evidence present next door. Channel-wise dropout, which zeroes whole feature maps, removes a detector outright and actually regularizes.

A warm restart lifts the learning rate back up, so validation worsens for several epochs by construction before recovering past the previous cycle's best. A patience of 5 reads that dip as failure. Judge once per cycle instead.

For a layer whose output is immediately normalized, rescaling its weights leaves the function unchanged. Weight decay there is not capacity control: it keeps the weight norm from growing, which stabilizes the effective step size those weights receive.

Not automatically. Smoothing tightens each class into an equidistant cluster in the penultimate layer and erases the similarity structure between classes, so it can raise top-1 accuracy while degrading nearest-neighbour retrieval built on the same features.

Decide by the depth you plan to reach, not by a shallow benchmark. Post-norm has been reported slightly better at around six blocks; pre-norm is the placement that still trains at a hundred. Trainability at the target depth dominates, and switching later invalidates your tuned hyperparameters.