How does an elementwise max such as ReLU route gradient in backpropagation?
answer
- one input wins, the other is ignored
- route, do not split
- the mask is 0/1, not a scaling
- the corner has no single slope
basics
~20 sA max passes the incoming gradient straight to whichever input won and gives exactly zero to the loser. For ReLU that is a 0/1 mask: positive inputs pass the upstream gradient through unchanged, non-positive ones cut it to zero.
solid answer
~50 sA max is a router, not a splitter. At each position it selects one input, so only that input can affect the output, and the whole upstream gradient goes to it while every other input gets exactly zero. `ReLU(x) = max(x, 0)` is the special case where one of the inputs is a constant: the derivative is 1 where `x > 0` and 0 where `x < 0`, so the backward pass is `dL/dx = G * mask`, an elementwise multiply by a 0/1 mask -- never a matrix product, because an elementwise op has a diagonal Jacobian. Note the zero is exact, not small. At exactly `x = 0` there is a kink and no derivative exists; implementations pick 0 or 1 by convention. The choice is empirically irrelevant, because both are valid subgradients and exact float equality with zero is vanishingly rare.
go deeper
Be able to state ReLU's derivative on both sides -- 1 for positive inputs, 0 for negative -- and that the backward pass multiplies the incoming gradient by that 0/1 mask elementwise.
Explain the general max rule behind it: the winning input receives the whole upstream gradient and the losing input receives an exact zero, in contrast to an addition, which copies the gradient to both operands.
Handle the edges cleanly. Know that the kink at zero has no derivative, that any convention in the interval from 0 to 1 is a valid subgradient, and why the choice has no measurable effect on training.
Own the consequences of hard routing as a design choice: a max gate assigns credit winner-take-all, so the gradient a branch receives depends on which branch currently wins, and that is a different optimisation landscape from any soft, differentiable weighting between branches.
## Max as a router Take an elementwise maximum of two inputs, `y = max(a, b)`, applied position by position. At each position the output equals one of the two inputs and completely ignores the other. Perturb the loser by a small amount and the output does not move at all; perturb the winner and the output moves one-for-one. So the local derivatives are `1` for the winner and `0` for the loser, and the backward pass sends the entire upstream gradient down the winning branch and nothing down the other. This is worth contrasting with the other two-input op everyone knows. An addition `y = a + b` *splits*: both operands have a local derivative of 1, so both receive a full copy of the upstream gradient. A max *routes*: one operand receives everything, the other receives an exact zero. Same arity, opposite gradient behaviour, and mixing them up is one of the more common whiteboard errors. A concrete case: suppose two branches each produce a score for the same position, and the network takes the elementwise max of the two score maps to combine them. Whichever branch produced the larger score at a position receives the full gradient there; the other branch receives nothing at that position and its parameters get no update signal from it. Over a batch each branch typically wins somewhere, so both still learn, but the credit at any single position is winner-take-all. ## ReLU is the one-sided special case `ReLU(x) = max(x, 0)`. The second operand is a constant, so "routing to the winner" collapses into a gate on the sign of the input: ``` dReLU/dx = 1 if x > 0 dReLU/dx = 0 if x < 0 ``` and the backward pass for a whole tensor is ``` dL/dx = G * mask, mask[i] = 1 if x[i] > 0 else 0 ``` where `G` is the upstream gradient and `*` is elementwise multiplication. Three properties are worth naming explicitly. **The zero is exact.** It is not "a small gradient" or "a slightly damped gradient". Where the input was non-positive, the gradient that continues backwards past that position is identically zero, and everything further upstream that only reached the loss through that position receives nothing from it. **The gradient is not scaled where it does pass.** ReLU's active-side derivative is exactly 1, so the upstream gradient goes through untouched. That is the whole point of a piecewise-linear gate compared with a squashing nonlinearity whose derivative is a fraction everywhere. **The mask comes from the forward pass.** You cannot know which positions were active without a record of the forward input's sign. Conveniently, for ReLU the *output* determines it too -- the output is positive exactly where the input was -- so the sign of either one suffices. ## Why the backward is a multiply, not a matmul An elementwise op maps position i of the input to position i of the output and touches nothing else. Its Jacobian is therefore diagonal: every off-diagonal entry is zero, because no input position influences any other output position. Multiplying by a diagonal matrix is the same as multiplying elementwise by its diagonal, so the backward of *any* elementwise nonlinearity is `dL/dx = G * f'(x)` -- one elementwise multiply, no matrix product, no transposes, and cost linear in the number of elements. This is the general template; the max is just the case where `f'` happens to be a 0/1 indicator rather than a smooth curve. ## The kink at exactly zero At `x = 0`, ReLU has a corner: the slope approaching from the left is 0 and from the right is 1, so no single derivative exists. The function is continuous but not differentiable at that one point. Implementations simply pick a value -- commonly 0, sometimes 1, occasionally 0.5 -- and the choice is a convention rather than a derivation. Why it does not matter in practice, and this is what an interviewer is checking: 1. **Any value in the interval from 0 to 1 is a valid subgradient** at the kink, because the function is convex there and each of those slopes supports it from below. Optimisation with subgradients is well behaved for convex kinks of this kind, so no choice in that range is "wrong". 2. **Exact equality with zero essentially never happens** with continuously distributed floating-point pre-activations. The set of inputs that hit the kink exactly has measure zero. The exceptions are deliberate: a tensor initialised to zeros, or an input that is already exactly zero, and even then the effect is one position's gradient. 3. **A single position's contribution is diluted** by the sum over the batch anyway. The wrong answer here is to claim the network is "not differentiable so gradient descent is invalid". Piecewise-linear networks are differentiable almost everywhere, and training operates on that almost-everywhere derivative without difficulty. ## Common misstatements to avoid - "The gradient is small for negative inputs" -- it is exactly zero. - "The backward multiplies by the input value" -- it multiplies by a 0/1 mask, not by `x`. - "Both branches of a max share the gradient" -- that is what an addition does; a max gives one branch everything. - "The tie case needs special handling to train correctly" -- it needs a chosen convention, and nothing more.
- What is the derivative of ReLU at exactly zero, and does the choice matter?None exists -- the left slope is 0 and the right slope is 1, so there is a kink. Implementations pick a convention, usually 0. It is empirically irrelevant: any value between 0 and 1 is a valid subgradient at a convex kink, exact float equality with zero almost never occurs for continuously distributed pre-activations, and one position's contribution is diluted by the batch anyway.
- Two expert branches produce scores for the same positions and the network takes their elementwise max. How does gradient reach each branch?Position by position, winner-take-all. At each position the branch with the larger score receives the full upstream gradient there and the other receives exactly zero, so a branch only learns from the positions where it currently wins. Across a batch both branches usually win somewhere and both keep training, but the routing is hard, not a soft weighting between them.
- Why is the backward pass of an elementwise nonlinearity a multiply rather than a matrix product?Because each input position affects only the matching output position, the Jacobian is diagonal -- every off-diagonal entry is zero. Multiplying by a diagonal matrix is identical to multiplying elementwise by its diagonal, so the backward is `dL/dx = G * f'(x)`: one elementwise multiply, cost linear in the number of elements, with no transposes and no contraction.
- How does a max differ from an addition in the way it distributes gradient to its two inputs?An addition splits and a max routes. Addition has a local derivative of 1 for both operands, so each receives a full copy of the upstream gradient. A max has local derivatives of 1 for the winner and 0 for the loser, so one operand receives everything and the other receives an exact zero. Same two inputs, opposite credit assignment.
saying these in an interview costs you the question
- Says ReLU's gradient for negative inputs is small rather than exactly zero
- Claims a max shares the gradient between both inputs
- Multiplies the upstream gradient by the input value instead of a 0/1 mask
- Says the kink at zero makes gradient descent invalid
- Thinks the backward of an elementwise op needs a matrix product