How do you derive the gradient of the loss L = ||Wx - y||^2 with respect to the matrix W?
answer
- name the residual first
- two stages, chained entrywise
- one weight touches one residual component
- the assembled result is an outer product
- gradient must have W's shape
basics
~20 sSet the residual r = Wx - y, so L = r transpose r. Entry W_ij affects only r_i, and does so through x_j, so the gradient is 2 (Wx - y) x transpose, a matrix with the same shape as W.
solid answer
~50 sIntroduce the residual `r = W x - y`, so `L = r^T r` and the derivative of L with respect to r is `2 r`. Now chain back to W. Each entry `W_ij` enters exactly one residual component, `r_i`, with `d r_i / d W_ij = x_j`, so `d L / d W_ij = 2 r_i x_j`. Assembling those entries gives the outer product `dL/dW = 2 (W x - y) x^T`. Two checks make the answer safe: the gradient must carry W's shape, and here `(W x - y)` is m by 1 while `x^T` is 1 by n, so the product is m by n; and setting m = n = 1 reduces the formula to `2 (w x - y) x`, the ordinary scalar answer. The gradient with respect to x is a different object, `2 W^T (W x - y)`.
go deeper
Be ready to recognise that L is a squared error built from a residual, and that differentiating a square brings down a factor of 2 times that residual.
An interviewer expects the entrywise derivation: show that W_ij enters only residual component i, with derivative x_j, and assemble those entries into an outer product.
Demonstrate the verification discipline — shape match, scalar collapse, zero-residual case, finite-difference spot checks — and distinguish the gradient with respect to W from the one with respect to x without hesitating.
Own the notational convention that a gradient carries the shape of the parameter it differentiates, and be able to explain why enforcing it across a team removes an entire class of transposition bugs.
## Setup and notation Let W be m by n, x be an n-vector, and y be an m-vector, so `W x - y` is an m-vector. The loss is the squared Euclidean norm `L = ||W x - y||^2 = sum over i of (r_i)^2`, where `r = W x - y`. The object we want, `dL/dW`, is by convention arranged to have the same shape as W: m by n, with entry (i, j) equal to `dL/dW_ij`. Committing to that shape rule up front is the single most effective guard against transposition errors. ## The chain, one stage at a time There are two stages: W produces the residual, and the residual produces the scalar loss. Stage two is easy. With `L = r^T r = sum r_i^2`, we get `dL/dr_i = 2 r_i`, so the gradient of L with respect to r is `2 r`. Stage one is where the bookkeeping lives. Component i of the residual is `r_i = (sum over k of W_ik x_k) - y_i`. Differentiate with respect to a single entry W_ij: - if the row index of the entry is not i, then `r_i` does not contain W_ij at all and the derivative is 0; - if the row index is i, only the k = j term survives, giving `d r_i / d W_ij = x_j`. So each weight entry touches exactly one residual component. That sparsity is what makes the final formula an outer product rather than something messier. ## Assembling the result Combining the stages entrywise, `dL/dW_ij = (dL/dr_i) * (d r_i / d W_ij) = 2 r_i x_j`. The matrix whose (i, j) entry is `r_i x_j` is exactly the outer product `r x^T`. Therefore `dL/dW = 2 (W x - y) x^T` with shape (m by 1)(1 by n) = m by n, the same shape as W. If your candidate answer does not have W's shape, it is wrong before any numbers are involved. ## Sanity checks worth doing out loud 1. Shape. As above: the gradient with respect to a matrix has that matrix's shape. 2. Scalar collapse. Put m = n = 1, so `L = (w x - y)^2`. The formula gives `2 (w x - y) x`, which matches the elementary one-variable answer. 3. Zero residual. If `W x = y` exactly, the gradient is the zero matrix — the loss is at its minimum value of zero and cannot be improved locally, which is what a zero gradient should mean. 4. Rank. The result is an outer product of two vectors, so it has rank at most one. That is expected: a single training example can only push W along one direction of the weight space. ## The other gradient Interviewers frequently follow up by asking for the derivative with respect to the input instead. Expanding `L = (W x - y)^T (W x - y)` and differentiating with respect to x gives `dL/dx = 2 W^T (W x - y)`, an n-vector. Note the transpose: `W` maps R^n to R^m, so its Jacobian is m by n, and pulling an m-dimensional sensitivity back to the n-dimensional input requires the transpose. Answering the W question with this expression, or the x question with the outer product, is the most common way this problem is failed. ## More than one example Stack N inputs as the columns of an n by N matrix X and the targets as an m by N matrix Y, and take the squared Frobenius norm `L = ||W X - Y||^2`, which is the sum of the per-example losses. The same derivation, applied term by term and summed, gives `dL/dW = 2 (W X - Y) X^T`, still m by n. The single-example formula is the N = 1 case, and the batch formula is a sum of N rank-one outer products, which is why a batch can move W in more directions than one example can. ## Verifying numerically When a matrix derivative has to be right, check a few random entries by finite differences: perturb one entry of W by a small amount in both directions, recompute the loss, and compare the central difference to the analytic entry. Agreement at several random positions is strong evidence; a mismatch that appears only on off-diagonal entries almost always means a transpose was dropped. ## How to answer Name the residual, differentiate the scalar loss with respect to it, differentiate the residual entrywise with respect to a single weight, note that only one residual component depends on that weight, assemble the outer product, and close with the shape check and the scalar collapse. That sequence is short, fully justified, and demonstrates the dimension discipline the question is really probing.
- What is the gradient of the same loss with respect to x rather than W?It is `2 W^T (W x - y)`, an n-vector. The residual sensitivity `2 (W x - y)` lives in the m-dimensional output space, and pulling it back to the n-dimensional input space requires multiplying by the transpose of the m by n Jacobian, which is W itself. Reporting the outer product here is the classic mix-up.
- How does the formula change for a batch of N examples stacked as columns?With inputs in an n by N matrix X, targets in an m by N matrix Y, and `L = ||W X - Y||^2` as the summed squared error, the gradient is `2 (W X - Y) X^T`, still m by n. It is a sum of N rank-one outer products, one per example, which is why a batch can move W in more directions than any single example can.
- How would you convince yourself a matrix gradient you just derived is correct?Check the shape against W, collapse to the scalar case and compare with the elementary answer, and confirm that a zero residual gives a zero gradient. Then verify a handful of random entries by central finite differences. Errors that show up only on off-diagonal entries are nearly always a dropped transpose.
saying these in an interview costs you the question
- Reports a gradient whose shape differs from W
- Gives 2 W transpose times the residual as the gradient for W
- Writes x times the residual transpose, transposing the outer product
- Drops the factor of 2 from the squared norm
- Claims every weight entry affects every residual component