Chain Rule & Curvature

## Definition Let `f: R^n -> R^m` be differentiable, with input `x = (x_1, ..., x_n)` and output components `f_1(x), ..., f_m(x)`. The Jacobian matrix of f at a point x is `J(x)[i][j] = d f_i / d x_j` evaluated at x and it has m rows and n columns. Say it as outputs-by-inputs: the row index runs over the things being produced, the column index over the things being varied. ## Why m by n and not n by m The defining property of the derivative is linear approximation. Near x, `f(x + d) ~= f(x) + J(x) d` where d is a small displacement in the input space, so d is a column in R^n, and the left-hand side lives in R^m. A matrix that eats an n-vector and returns an m-vector has m rows and n columns. That is the whole argument, and it is the one to give in an interview: the shape is not a convention to memorise, it is whatever makes the product conformable. If you ever cannot remember the orientation, reconstruct it from `J d` in half a second. Row i of J is `(d f_i / d x_1, ..., d f_i / d x_n)` — the gradient of the scalar-valued component f_i laid out horizontally. Column j is the vector of how every output responds to nudging the single input x_j. ## A worked example Take `f: R^2 -> R^2` with `f(x_1, x_2) = (x_1^2 * x_2, x_1 + sin(x_2))`. Then `J = [[2 * x_1 * x_2, x_1^2], [1, cos(x_2)]]` Row 1 differentiates `x_1^2 * x_2` with respect to x_1 and then x_2; row 2 does the same for `x_1 + sin(x_2)`. Here n = m = 2, so the matrix is square — a special case, not the general one. A second example fixes the shape rule in memory: `f: R^5 -> R^3` given by `f(x) = A x + b` with A of shape 3 by 5. Every partial derivative is a constant entry of A, so `J(x) = A` everywhere. An affine map is its own linearization, and its Jacobian is exactly the matrix of the map — 3 by 5, m by n, as required. ## The scalar-output special case When m = 1, f is a scalar field such as a loss, and the Jacobian is a single row of length n. The gradient is conventionally a column vector, so `gradient = J^T`. Keeping these straight avoids a lot of transposition pain: Jacobians are wide (outputs by inputs), gradients of scalars are tall. Many disagreements about a stray transpose are really disagreements about which of these two objects someone wrote down. ## It is a function of the point The Jacobian is not one matrix attached to f; it is a matrix-valued function of x. In the worked example above, J at (1, 0) is `[[0, 1], [1, 1]]` while J at (0, 1) is `[[0, 0], [1, cos(1)]]`. Any statement about a Jacobian that does not say where it is evaluated is incomplete, and forgetting the evaluation point is the error that ruins multi-stage chain-rule computations, where each stage is evaluated at a different intermediate value. ## Layout conventions What is described here is the numerator layout: outputs index rows. Some texts use the transposed denominator layout, in which the same object appears as n by m. Neither is wrong, but mixing them inside one derivation produces transposes that appear from nowhere. The practical discipline is to declare the convention once, then verify every result by shape: if a quantity is a derivative of something m-dimensional with respect to something n-dimensional, and your expression is not m by n, something is wrong before any number is computed. ## Why it matters downstream Every composed derivative is built from these blocks. Once the shape rule is automatic, checking a long derivation reduces to checking that adjacent matrices are conformable and that the ends match the overall input and output dimensions. That single habit catches most sign-free algebra errors without evaluating anything. ## Answering well A strong answer states the shape, gives the entry formula, justifies the shape from the linear approximation rather than from memory, notes that rows are component gradients, and adds that the Jacobian is evaluated at a point. Mentioning the m = 1 case and the transpose relationship to the gradient shows the fluency an interviewer is listening for.