Which automatic differentiation mode builds the Jacobian of thousands of residuals against 6 robot-arm parameters in fewer sweeps?
answer
- the shape of the Jacobian decides
- count inputs against outputs
- six columns, thousands of rows
- one forward sweep fills one column
basics
~10 sForward mode. One forward sweep produces one Jacobian column, so six sweeps cover the six parameters. Reverse mode produces one row per sweep and would need thousands, one per residual.
solid answer
~50 sThe residual Jacobian here is thousands of rows by 6 columns. A forward sweep seeded with the basis direction for parameter `i` returns column `i` — the sensitivity of every residual to that parameter — so six sweeps fill the whole matrix. A reverse sweep returns one row, the derivative of a single residual with respect to all six parameters, so you would need one sweep per residual: thousands. Forward mode wins by the ratio of rows to columns, and it stores nothing from the primal pass because tangents travel alongside values. The caveat matters in interviews: if the algorithm only needs the gradient of the summed squared error, that is one scalar output and one reverse sweep does it. Forward mode wins specifically because Gauss-Newton style updates want the residual-wise Jacobian, not just a gradient.
go deeper
Know that two directions of automatic differentiation exist and that forward mode propagates derivatives alongside values in the same pass, rather than in a second backward pass.
Be able to state the counting rule both ways: a forward sweep yields one Jacobian column, a reverse sweep one row, so the cheaper mode is decided by which dimension is smaller.
Show the judgment step — ask what the solver consumes before picking a mode, and notice when a full Jacobian is wanted rather than a scalar gradient. Mention that forward mode keeps no intermediate state.
Own the cost model for a whole team: which derivative shapes your workloads need, whether an algorithm can be reformulated to use matrix-free products, and when paying for both modes is justified.
## The setup Calibrating a robot arm means fitting a small parameter vector — link lengths and joint offsets, say `theta` in `R^6` — so that predicted marker positions match measured ones. Each measurement contributes a residual, and there are thousands of them, giving `r(theta)` in `R^m` with `m` in the thousands. The Jacobian `J = dr/dtheta` is `m x 6`: **very tall and very thin**. Classical least-squares solvers want that matrix. A Gauss-Newton step solves `(J^T J) d = -J^T r` for the update `d`; Levenberg-Marquardt adds a damping term to the same normal equations. Both need `J`, or at least products with it. ## Counting sweeps The rule is fixed by which vector each mode lets you seed: - A **forward** sweep computes `J u` for a chosen input direction `u`. Seed `u = e_i` and you get **column `i`**: how all thousands of residuals move when parameter `i` moves. Six columns, **six sweeps**. - A **reverse** sweep computes `v^T J` for a chosen output direction `v`. Seed `v = e_j` and you get **row `j`**: how residual `j` responds to all six parameters. Thousands of rows, **thousands of sweeps**. Each sweep costs a small constant multiple of one evaluation of `r`, so forward mode is cheaper by roughly `m / 6` — two to three orders of magnitude here. This is the mirror image of network training, where a hundred million inputs and one output make reverse mode the only viable direction. Nothing about the modes changed; the shape of the function did. ## How the forward sweep does it Forward mode carries a tangent next to every intermediate value and updates it with the operation's local rule at the same moment the value is produced: for `z = g(x, y)`, `dz = (dg/dx) * dx + (dg/dy) * dy`. Seeding the six parameter tangents with `e_i` and running the kinematic chain once yields the tangent of every residual, which is exactly column `i`. Because value and tangent are produced and consumed together, **nothing from the primal pass has to be kept alive** — extra memory is a constant factor on live values, not a growing store. On a long simulation loop that property alone can decide the mode. ## The caveat a good candidate raises unprompted If your optimiser only needs the gradient of the scalar objective `L = 0.5 * ||r||^2`, forward mode is *not* the right answer: `L` is one output, so one reverse sweep returns all six partials, versus six forward sweeps. And there is a sharper trick — the reverse seed need not be a basis vector. Seeding the reverse sweep with the residual vector `r` itself returns `r^T J`, which is `(J^T r)^T`, the gradient of the sum of squares, in **one** sweep and without ever forming `J`. So the honest answer is conditional on the algorithm: plain gradient descent on the sum of squares wants one reverse sweep; Gauss-Newton and Levenberg-Marquardt want the residual-wise `J` (or repeated products with `J` and `J^T`), and that is where six forward sweeps beat thousands of reverse ones. ## The other shape that favours forward mode Directional sensitivity questions have the same profile. Ask of a control policy: *if the state is perturbed along direction `u`, how much does the commanded action move?* That is `J u` for one specific `u` — one forward sweep, no Jacobian formed, cost about one policy evaluation. Answering it by reverse mode would take one sweep per action component, then a matrix-vector product you did not need. Composing the modes is also useful: running forward mode over a reverse-mode gradient gives a Hessian-vector product at the cost of a few sweeps, again without forming the `n x n` matrix. ## The decision procedure 1. Write down what the algorithm actually consumes: a gradient, a full Jacobian, or products with a Jacobian. 2. Count the derivative directions that requires — input directions for forward, output directions for reverse. 3. Pick the mode whose count is smaller; if both counts are large, ask whether the algorithm can be reformulated to use matrix-free products instead of the explicit matrix. ## What an interviewer is checking That 'always use reverse mode' is a habit you can justify and therefore also override. Recognising the tall-and-thin shape, and knowing that the seed vector is yours to choose, is what separates someone who has only trained networks from someone who understands differentiation as a cost model.
- If you only needed the gradient of the summed squared error, would forward mode still be the cheaper choice?No. The sum of squares is a single scalar output, so one reverse sweep returns all six partials while forward mode would need six. Better still, seeding the reverse sweep with the residual vector itself returns `J^T r` — the gradient — in one sweep, without forming the Jacobian at all.
- How would you get the sensitivity of a control policy's action to one perturbation direction of its input?One forward sweep seeded with that direction returns the Jacobian-vector product directly: the change in every action component along that direction, for about the cost of one policy evaluation. No Jacobian is formed and no reverse sweep is needed, which also makes it cheap to repeat for a handful of directions.
- Why does forward mode need no storage of intermediate values while reverse mode does?Forward mode computes each tangent at the same moment as its value, so both are consumed immediately and nothing has to survive. Reverse mode is two phases: the backward phase reads quantities produced during the forward phase, so those must stay alive in between, and memory grows with the length of the computation.
saying these in an interview costs you the question
- Says reverse mode is always cheaper because deep learning uses it
- Claims one reverse sweep can produce the whole residual Jacobian
- Thinks a forward sweep returns a row rather than a column
- Counts sweeps by network depth instead of inputs or outputs
- Never asks whether the algorithm needs the Jacobian or just a gradient