Why does dropout reduce overfitting in a fully connected network?

## What dropout actually does Dropout is a training-time perturbation applied to a layer's outputs. For each training forward pass you draw, independently for every unit in the layer, a Bernoulli variable that is 1 with the *keep probability* `keep = 1 - p` and 0 otherwise, and multiply the unit's activation by it. Units that draw a 0 output zero for that step and receive no gradient for that step; the surviving units carry the whole forward and backward pass. A fresh mask is drawn every step (in practice, independently per example in the batch), so the pattern of who is present changes constantly. Nothing is removed from the model permanently. The weights of a dropped unit are untouched — they simply get no update from that step — and every unit is back in play with probability `keep` on the next one. ## Explanation one: it breaks co-adaptation In a network trained without noise, hidden units are free to form fragile joint arrangements: unit A can learn a feature that is only correct when unit B is also firing, with B quietly compensating for A's error. Such an arrangement fits the idiosyncrasies of the training set well and generalises badly, because it depends on a precise conspiracy of units rather than on a feature that is independently meaningful. Dropout makes that conspiracy unreliable. A unit that leans on a specific partner is wrong every time the partner is dropped, and gradient descent pushes it toward features that pay off across many random contexts. The learned representation ends up more redundant and more distributed: several units carry overlapping evidence for the same concept, which is exactly the property that survives a distribution shift between training and test data. ## Explanation two: implicit ensembling With `n` units eligible for dropping there are `2^n` distinct masks, hence `2^n` possible thinned subnetworks. Every training step samples one of them and updates the weights that subnetwork uses. Because all subnetworks share one parameter tensor, a single training run trains all of them at once, with each individual mask seen at most a handful of times. This is why dropout is described as training an exponentially large ensemble at the cost of one model. An ensemble is only useful if you can average it. Explicitly sampling many masks at evaluation and averaging the predictions would be expensive, so dropout instead uses a *weight-scaling inference rule*: run the full, unmasked network once, with activations scaled so the expected input to each unit matches what it saw in training. That single deterministic pass approximates the ensemble's averaged prediction. For a linear model with a softmax output the approximation is exact; for a deep nonlinear network it is an approximation that works well in practice. ## The rate is a dial, not a constant The drop rate trades capacity for regularisation. At rate 0 you have no dropout; as the rate rises, fewer units are present per step and the network is pushed harder toward redundant features; near 1 nothing can be learned. A rate around 0.5 on a wide fully connected layer is the classic setting, with smaller rates on narrow layers and on inputs, where zeroing half the signal destroys too much. Because dropout removes active capacity, raising the rate often has to be paired with widening the layer. Read the two failure directions off the loss curves. Too much dropout: training loss stalls at a high value and the gap between training and validation loss collapses toward zero — the model is underfitting. Too little: training loss keeps falling while validation loss bottoms out and turns up. Dropout is not free. Gradients are noisier, so runs take more epochs to converge and the validation curve is bumpier from epoch to epoch. It buys the most when parameters are plentiful relative to data — wide classifier heads on small datasets — and much less when data is abundant. ## What it is not Dropout adds no term to the loss. It changes *which function* you are optimising: instead of minimising the loss of one fixed network you minimise the expected loss over random masks. That is why it is grouped with the penalty-free regularisers rather than with weight decay, and why you will not see it appear anywhere in the objective you print during training.