ReLU and Its Variants

## What the unit computes Each hidden unit first forms a **pre-activation** `z = w·x + b` — a weighted sum of its inputs plus a bias — and then passes it through a nonlinearity. ReLU, the *rectified linear unit*, is that nonlinearity in its simplest useful form: ``` f(z) = max(0, z) ``` Positive pre-activations come through untouched; negative ones are clamped to exactly zero. Graphically it is a hinge at the origin: a flat segment along the negative axis, then a 45-degree line. ## The derivative, which is the reason it exists ``` f'(z) = 1 for z > 0 f'(z) = 0 for z < 0 ``` Backpropagation forms a unit's error signal by multiplying the error arriving from above by that local derivative. With a slope of exactly 1, an **active** unit neither shrinks nor amplifies what passes through it — the activation is transparent, and the only rescaling of the backward signal comes from the weight matrices. An activation whose derivative is always well below one does the opposite: it multiplies in a factor smaller than one at every layer, and those factors compound with depth. Removing that per-layer shrink on the active path is the single mechanical reason ReLU let people train much deeper stacks than the earlier squashing units allowed. ## Piecewise linear, but not linear A common misreading is that ReLU is "basically linear", so a stack of them collapses into one linear map. It does not. For a *fixed input*, the set of active units is fixed, and on that region the whole network is an affine function — but the active set changes as the input moves. The network is therefore **piecewise linear**: the input space is carved into regions, each with its own affine map, and the boundaries between regions are learned. That is a genuinely nonlinear function class, and it is expressive enough for the usual universal-approximation results. ## Sparsity For any given input, a substantial fraction of units sit on the flat branch and emit exact zeros. This is often sold as a feature, and there is a real representational argument: different inputs recruit different subsets of units, so the representation is conditional rather than uniformly dense. What it is *not* is free speed. A dense matrix product multiplies and accumulates the zeros like any other number; you only save arithmetic if something explicitly skips them, and unstructured, input-dependent zeros are hard to exploit. Treat sparsity as a property of the representation, not as an optimisation. ## The costs you sign up for 1. **Non-negative outputs.** Every value a ReLU layer emits is at least zero, so the next layer's inputs have a strictly positive mean. That shifts its pre-activations, and it means all of the incoming weights of a downstream unit receive gradients sharing the sign of that unit's upstream error, so the weight vector tends to move in one direction at a time rather than freely. Normalisation layers, or a nonlinearity that can emit negative values, offset it. 2. **Unbounded above.** Nothing caps the activation, so bad scaling upstream shows up as very large forward values rather than being squashed. 3. **A flat negative branch.** A unit whose pre-activation is negative on *all* data emits zero everywhere and receives exactly zero gradient — the dead-unit failure, which is the price of the clean positive-branch derivative. 4. **A kink at zero.** `f` is not differentiable at `z = 0`. Any value in the interval from 0 to 1 is a valid subgradient there and a fixed convention is simply chosen; a pre-activation that is exactly zero in floating point is vanishingly rare, so the choice has no practical effect. ## How to answer the "why default" part Because it is the cheapest thing that works: one comparison per element, no transcendental function, a derivative of exactly one where the unit is active, and no upper saturation to stall learning on that branch. Later nonlinearities smooth the kink or reshape the negative branch, and they buy small, workload-dependent improvements — but they all cost more per element, and ReLU remains the reasonable starting default that you deviate from with evidence.