What are the gradients of w^T x and of x^T A x with respect to the vector x?

## Why these two forms A huge share of the objectives that appear in modelling and optimisation are built from exactly two pieces: a **linear form** `w^T x`, a weighted sum of the entries of `x`, and a **quadratic form** `x^T A x`, a weighted sum of all pairwise products of entries. Knowing their gradients cold means you can differentiate most objectives without dropping to index notation each time. Interviewers ask it to see whether you understand vector differentiation or merely memorised two formulas. ## Conventions first Take `x` to be a column vector with `n` entries, `w` a constant column vector with `n` entries, and `A` a constant `n`-by-`n` matrix. Both `w^T x` and `x^T A x` evaluate to a single number. The gradient of a scalar with respect to `x` is the vector of partial derivatives, one per entry of `x`, so it has the same shape as `x`. That shape rule is the fastest error check available: if your candidate answer is a matrix or a scalar, it is wrong before you check any arithmetic. ## The linear form Expanded, `w^T x = w1*x1 + w2*x2 + ... + wn*xn`. Differentiate with respect to a single entry `xk`. Every term except the `k`-th is constant in `xk` and contributes zero; the `k`-th term is `wk * xk`, contributing `wk`. So the `k`-th component of the gradient is `wk`, for every `k`, and therefore `grad_x (w^T x) = w` The scalar analogue is `d/dx (w*x) = w`. The gradient is constant — the same vector at every point — which is exactly what 'linear' means: the rate of change does not depend on where you are. ## The quadratic form Expanded with indices, `x^T A x = sum_i sum_j xi * A_ij * xj`. Differentiate with respect to `xk` and ask which terms survive. 1. Terms where `i = k` and `j` is anything other than `k`: each is `xk * A_kj * xj`, contributing `A_kj * xj`. Summing over `j` gives the `k`-th entry of `A x`. 2. Terms where `j = k` and `i` is anything other than `k`: each is `xi * A_ik * xk`, contributing `A_ik * xi`. Summing over `i` gives the `k`-th entry of `A^T x`. 3. The single term where `i = j = k` is `A_kk * xk^2`, whose derivative is `2 * A_kk * xk` — which is precisely one contribution from each of the two counts above, so it is already included. Adding the pieces, the `k`-th component of the gradient is `(Ax)_k + (A^T x)_k`, hence `grad_x (x^T A x) = (A + A^T) x` ## The symmetric case If `A` is symmetric, meaning `A^T = A`, the two contributions coincide and `grad_x (x^T A x) = 2 A x` This is the version most often quoted, because the matrices appearing in quadratic objectives are very frequently symmetric by construction. But quoting `2Ax` for a general `A` is a genuine error, and interviewers plant asymmetric matrices to catch it. The safe habit is to derive `(A + A^T)x` and then say 'which is `2Ax` because `A` here is symmetric'. A related fact worth knowing: only the symmetric part of `A` affects the value of `x^T A x` at all, because the antisymmetric part contributes zero for every `x`. That is the deeper reason the gradient depends on `A` only through `A + A^T`. ## Special cases and checks **Sum of squares.** Take `A` to be the identity matrix. Then `x^T x = x1^2 + x2^2 + ...`, and the formula gives `grad = 2x`. That matches the direct computation: the partial with respect to `xk` is `2*xk`. **One dimension.** With `n = 1`, `w^T x` is `w*x` with derivative `w`, and `x^T A x` is `a*x^2` with derivative `2*a*x`. Both formulas reduce correctly, which makes the scalar case a reliable memory aid. **Shape check.** `A` is `n`-by-`n` and `x` is `n`-by-1, so `(A + A^T)x` is `n`-by-1, matching `x`. Any answer of the wrong shape — say, `A` alone, or `x^T A` — fails immediately. **A trap.** The reflex 'power rule' answer `A x` for the quadratic form drops one of the two ways `xk` enters the double sum. Explaining *why* the factor of two appears — `x` occurs on both sides of `A` — is the part that distinguishes a derived answer from a recalled one. ## What good answers sound like A strong candidate writes both results, names the symmetry condition explicitly, sketches the index argument for the factor of two in under a minute, and verifies with the scalar case. A weak one recites `w` and `2Ax` and cannot say what happens when `A` is not symmetric.