What does inverted dropout divide surviving activations by during training?

## The arithmetic Let `a` be one unit's activation, `p` the drop rate, and `keep = 1 - p` the keep probability. Inverted dropout computes, at training time, ``` m ~ Bernoulli(keep) out = a * m / keep ``` and at evaluation time simply ``` out = a ``` Take the expectation over the mask: `E[out] = keep * (a / keep) + (1 - keep) * 0 = a`. The training-time layer therefore agrees *in expectation* with the evaluation-time layer, which is why no adjustment is needed when you switch modes. Without the division the expectation would be `keep * a` — at a drop rate of 0.5 every downstream layer would suddenly see inputs twice as large at evaluation as the ones it was fitted against. ## The worked case Take a 512-unit hidden layer at drop rate 0.5, and suppose for clarity every activation equals 1. Each unit survives with probability 0.5 and, when it survives, is divided by 0.5, i.e. doubled to 2. The expected number of survivors is 256, each contributing 2, so the expected sum is 512 — the same as the unmasked layer's sum of 512. The match is only in expectation. The realised sum fluctuates: each masked-and-scaled unit has variance `(1 - keep) / keep`, which is 1 at `keep = 0.5`, so the sum over 512 units has variance 512 and a standard deviation of about 23, roughly 4 to 5 percent of the mean. Wide layers average this noise down; narrow layers do not, which is one reason high rates on small layers destabilise training. ## Why the expectation is matched but the variance is not The second moment tells a different story. `E[out^2] = keep * (a / keep)^2 = a^2 / keep`, so the per-unit second moment is inflated by a factor `1 / keep` relative to the unmasked activation, and the variance contributed by the mask is `a^2 * (1 - keep) / keep`. At evaluation that extra variance vanishes entirely, because there is no mask left. This is the single most important consequence of inverted scaling, and it is what makes dropout interact badly with any downstream layer whose behaviour depends on the *scale* of its inputs rather than only their mean. Matching the first moment is enough for a plain linear layer; it is not enough for a layer that estimates and stores the variance of what it sees. ## Inverted versus the original scheme The original formulation left training untouched — mask, no rescale — and multiplied the weights by `keep` at test time so that each unit's expected input matched training. Both schemes make the two modes agree; they differ in where the constant lives. Inverted scaling is preferred because it keeps the inference-time computation exactly equal to that of an ordinary network. Three practical consequences follow. You can use different rates in different layers without carrying a table of per-layer scale factors into deployment. You can anneal or schedule the rate during training and the evaluation path never changes. And a model exported for serving has no dropout-shaped residue in it at all, so nothing about serving depends on how the model was regularised. ## Common mistakes Dividing by the *drop* rate instead of the keep probability inverts the correction and blows up the activations at high rates. Multiplying by `keep` during training rather than dividing doubles the shrinkage instead of undoing it. Applying the scale to dropped units as well is a no-op, since they are already zero. And scaling the *loss* rather than the activations does not restore anything, because the mismatch lives in the forward pass of every downstream layer, not in the objective's magnitude. ## What to say in an interview State the rule (`divide survivors by 1 - p`), give the one-line expectation calculation that justifies it, name the payoff (an untouched evaluation path), and mention that only the mean is preserved — the variance is inflated by `1 / keep`. That last sentence is what separates a memorised rule from an understood one.