Why does elementwise dropout barely regularize a convolutional feature map?

## The correlation problem A convolutional layer slides one filter over its input, so the value at position `i` of a feature map and the value at position `i + 1` are computed by the same weights from receptive fields that overlap almost completely. In a 1D convolution over an ECG signal with a kernel spanning, say, 15 samples and a stride of 1, two adjacent outputs share 14 of their 15 inputs. Their activations are close to duplicates. Dropout's regularising force comes from making a piece of information genuinely unavailable, so that whatever depends on it has to survive without it. When the masked value is nearly reproduced by its neighbour, that force evaporates. Zero position `i` and the evidence is still sitting at `i - 1` and `i + 1`; the next layer, whose own receptive field spans many positions, aggregates over the survivors and barely notices. You have injected noise — which slows convergence and makes the validation curve bumpier — without removing anything the network relied on. The same argument is why the effect is worst in early layers with large spatial extent and heavy overlap, and less pronounced deep in the stack where activations have been repeatedly downsampled and are less redundant. ## Changing the unit that gets dropped The fix is not a different rate but a different *unit of dropping*. Channel-wise dropout, usually called spatial dropout, samples one Bernoulli variable per channel rather than per element, and zeroes that channel's entire feature map together. Because a channel is one learned filter's response, dropping it removes a whole detector for that step: whatever the network learned to do with the output of the eleventh filter must be doable without it. That is a real, non-recoverable perturbation, and it is what makes co-adaptation between filters costly. Practical notes that follow from the coarser unit. The same inverted scaling applies — survivors are divided by the keep probability — but the mask has one entry per channel instead of one per activation, so the noise is far coarser and the gradient variance per step is higher. Rates are usually kept lower than the classic setting for wide fully connected layers, because dropping half the channels of a 64-channel layer removes a large fraction of the model's representational width in one step. And with few channels the coarse mask can be too blunt to use at all. ## A coarser unit still: stochastic depth The same idea taken one level up drops entire residual blocks. In a 110-block residual stack, stochastic depth assigns each block a survival probability and, when a block is dropped for a training step, the block's residual branch is skipped entirely and the block computes the identity — the input passes through the skip connection unchanged. The network stays valid because the skip path always exists; you have simply trained a shallower network for that step. Two details matter. Survival probability is typically decayed linearly with depth, so early blocks that everything downstream depends on are kept almost always while late blocks are dropped often; this reflects that late blocks make smaller, more specialised refinements. And at evaluation all blocks are used, with each block's residual branch weighted by its survival probability so that the expected contribution matches training. Beyond regularisation this shortens the average backward path, which helps very deep stacks train, and it cuts training time because dropped blocks are never computed. ## The recurrent case Recurrent layers show the same lesson from a different angle. If you resample an independent mask at every time step of a sequence, the noise applied to the hidden state compounds step after step and the state's ability to carry information over long spans is destroyed — the model degrades rather than regularises. Sharing one mask across all time steps of a given sequence, and drawing a fresh one for the next sequence, keeps the perturbation consistent along the sequence: the same units are absent throughout, so the recurrence can still transport what survives. On a part-of-speech tagger, per-step resampling erases exactly the long-range context the recurrence exists to provide. ## The unifying rule Drop at a granularity coarse enough that what you removed cannot be reconstructed from what remains, and consistent along whatever axis the model uses to accumulate information. Elementwise masks are the right choice for a fully connected layer, where units are genuinely separate features; they are the wrong choice wherever the architecture has deliberately built in redundancy or recurrence.