Why does a stack of fully-connected layers with no nonlinearity collapse to one layer?

## The algebra A fully-connected layer without an activation computes `y = W x + b` for a weight matrix `W` and a bias vector `b`. This is an *affine* map. Compose two of them: ``` y = W2 (W1 x + b1) + b2 = (W2 W1) x + (W2 b1 + b2) ``` Define `W' = W2 W1` and `b' = W2 b1 + b2`. The composite is `y = W' x + b'` — one affine map. The argument applies again to `W' ` and a third layer, and by induction to any depth: an arbitrarily deep stack of activation-free layers is one affine map from input to output. This is a statement about the *set of representable functions*, and it is exact, not approximate. Depth here is not a weak improvement; it is zero improvement in expressive power. ## Why the biases do not save it A common wrong answer is that biases introduce nonlinearity. They do not. A bias shifts, and shifts compose into shifts: the composite bias is `W2 b1 + b2`, still just a vector added at the end. The map remains affine, its level sets remain flat, and its decision boundary — if you thresholded the output — remains a hyperplane. ## Why width does not save it either Making the hidden layers wider adds parameters but changes nothing about the function class, because the product `W2 W1` is still a single matrix of the input-to-output shape. Worse, going *narrow* actively hurts. If `W1` maps 100 inputs to 5 hidden units and `W2` maps those 5 to 100 outputs, then `rank(W2 W1) <= 5`. The composite is not merely an affine map; it is a *rank-constrained* affine map. A single unrestricted layer of the same input/output widths can express strictly more. This is exactly why low-rank factorisation is a legitimate compression trick — you deliberately trade capacity for parameters — and exactly why it is not a source of capacity. ## The symptom in practice The canonical way this shows up: someone builds a three-layer network for a two-input parity task — output 1 when exactly one of two binary flags is on — trains it carefully, tunes the learning rate, widens the hidden layers, and reports that accuracy sits at chance-like 50–75% and refuses to move, matching a plain linear model exactly. The instinct is to add depth or capacity. The correct diagnosis is that no nonlinearity is applied between the layers, so the whole network *is* a linear model; the target function is not affine, and no setting of the parameters can reach it. Diagnostics that confirm it quickly: multiply the learned weight matrices together and check that the product reproduces the network's outputs on held-out inputs; or fit a single linear layer and observe that it matches the deep model's loss almost exactly, run after run. ## Where the nonlinearity must go Between *consecutive* linear maps. A frequent half-fix is to apply a nonlinearity only at the output — for example squashing the final score into a probability. That leaves the entire hidden body collapsible: the network is one affine map followed by one squashing function, which is a generalised linear model, not a deep one. Its decision boundary is still a hyperplane in the input space. Every hidden layer needs its own nonlinearity for depth to mean anything. ## The nuance worth knowing Identical function class does not mean identical training. A deep linear stack is a different *parameterisation* of the same function set, and reparameterisation changes the loss surface's geometry, the conditioning of the problem and therefore the trajectory that gradient descent takes. Deep linear networks are studied precisely because they have non-trivial training dynamics while having trivial expressive power. But the ceiling is the ceiling: whatever path training takes, the endpoint is an affine map, so a task that requires a curved decision boundary remains out of reach. ## How to answer the interview version State the composition identity with the explicit composite weight and bias. Say the word *affine*, and note that biases compose into a bias. Add the rank observation about narrow middle layers to show you understand the collapse can be a restriction rather than a wash. Close with the operational point: the fix is a nonlinearity between every adjacent pair of linear maps, not more layers and not more units.