When would you replace ReLU with Leaky ReLU, PReLU, or ELU, and what does each cost?

## The shared shape Every member of the family agrees on the positive branch: `f(z) = z` for `z > 0`, derivative 1. That is the part of ReLU nobody wants to change, because it is what keeps the backward signal from being scaled down by the activation. All the design work happens below zero, where plain ReLU outputs 0 and has derivative 0 — the flat region that both produces exact-zero sparsity and makes permanent silence possible. ## Leaky ReLU — a fixed crack in the door ``` f(z) = z for z > 0 f(z) = 0.01 * z for z <= 0 (0.01 is the conventional slope) ``` The derivative below zero is the slope itself rather than 0. A unit whose pre-activation has been pushed negative still receives a small but nonzero gradient, so the loss can lift it back. Cost per element is essentially the same as ReLU — a select between two linear expressions, no transcendental function. What you give up: **exact zeros**. Leaky outputs are small, but almost never precisely 0.0, so the activation tensor is dense. If you chose ReLU because a downstream step, an interpretability story or a structural argument depended on genuine zeros, a leak removes that. Note also that the negative branch is still unbounded — a very negative pre-activation produces a very negative output, just scaled down. ## PReLU — let the model choose the slope PReLU has the same two-piece shape, but the negative slope is a **learned parameter**, trained by gradient descent alongside the weights. The usual granularity is one slope per channel (sometimes one per layer), so a convolutional layer with 64 channels adds 64 parameters — negligible against the weight count, but not free. Why bother: the right amount of leak is not obviously the same in every layer or channel, and letting the model decide removes a hyperparameter. PReLU is a familiar choice in image super-resolution stacks, where preserving fine low-magnitude detail through many layers matters and hard rectification throws information away at every step. What the learned values tell you is genuinely useful diagnostic information. **Slopes settling near zero** mean those channels want hard rectification — the data is not asking for a leak, and the discarded negative half is doing real work as a gate; you are paying parameters and an extra elementwise multiply for behaviour plain ReLU would have given you. **Slopes drifting toward one** mean the opposite: the unit is close to the identity map, so the nonlinearity is contributing little there, which is worth a second look at whether that layer is earning its place. On small datasets the extra parameters are a mild overfitting risk, and they interact with weight decay in ways that are easy to get wrong if the slopes are lumped in with the weights. ## ELU — saturate instead of leak ``` f(z) = z for z > 0 f(z) = alpha * (exp(z) - 1) for z <= 0 ``` As `z` goes far negative, `exp(z)` goes to 0 and `f(z)` approaches `-alpha`: the negative branch **saturates at a bounded value** rather than continuing linearly downward. Two consequences follow. First, the unit can emit negative outputs, so the layer's mean activation is pulled toward zero instead of sitting strictly positive — that removes the mean shift a rectified layer imposes on the next layer's inputs. Second, extreme negative pre-activations map to a bounded value rather than an extreme one, which makes the representation less sensitive to outliers in the pre-activation. The costs are real. The derivative below zero is `alpha * exp(z)`, which decays toward 0 as `z` becomes very negative — so ELU does not remove quiet units so much as make the quieting smooth and bounded rather than a cliff at exactly zero. And an exponential per element is meaningfully more expensive than a comparison, in both training and inference, which matters more than it sounds when the activation is applied to every element of every feature map. ## How to actually choose Start with ReLU. It is the cheapest, and the differences between these variants are usually small and workload-dependent — small enough to be swamped by learning-rate and regularisation choices. Switch when you have a reason you can state: - Units are going silent and you would rather make the failure structurally impossible than depend on a well-behaved learning-rate schedule: take the fixed leak. - The right leak plausibly varies across the network, the parameter budget is irrelevant next to the weights, and you want the diagnostic signal the learned slopes give you: take PReLU. - You want negative, mean-centred, bounded activations and can afford an exponential on every element: take ELU. And be honest about the order of operations. If units are dying because the learning rate is wrong, changing the activation hides a problem you should have fixed; the variant is insurance, not a substitute for a sane optimisation setup.