Why does training a 100-million-parameter network use one reverse sweep instead of one forward sweep per parameter?
answer
- count sweeps, not parameters
- one sweep per output, or per input
- a scalar loss is one output
- cotangent seeded with 1, propagated backwards
basics
~20 sReverse mode costs one sweep per output; forward mode costs one per input. Training has one scalar loss and a hundred million inputs, so a single reverse sweep gets every gradient, while forward mode would need a hundred million.
solid answer
~50 sAutomatic differentiation propagates derivatives through the graph of primitive operations in one of two directions. Forward mode carries a tangent alongside each value — one Jacobian-vector product per sweep — so one sweep gives the derivative of every output with respect to one chosen input direction, and its cost scales with the number of inputs. Reverse mode carries a cotangent backwards through a chain of vector-Jacobian products, so one sweep gives the derivative of one output with respect to every input, and its cost scales with the number of outputs. Training has exactly one output: the scalar loss. Seeding it with 1 and running one reverse sweep yields all 100 million partial derivatives for roughly twice the work of a forward pass, independent of parameter count. The price is memory: values from the forward phase must stay alive until the backward phase consumes them.
go deeper
Be ready to state that one backward pass produces every parameter's gradient at once, and that the number of passes does not grow when the network gets bigger.
Explain the counting rule out loud: one reverse sweep per output, one forward sweep per input, and show why a single scalar loss makes reverse mode the obvious direction here.
Expect to price the choice in a real run — roughly what fraction of step time the backward sweep takes, and that reverse mode buys its cheap gradient by keeping forward-phase values alive.
Own the framing that a gradient costs a constant multiple of a function evaluation. Use it to budget training compute and to challenge any proposed method that needs derivatives along many separate input directions.
## The quantity in question A network with its loss attached is a function `f` from `n` inputs to `m` outputs. During training the inputs are the parameters (100 million of them) and there is exactly one output, the scalar loss, so `m = 1`. The object that holds all the derivatives is the Jacobian: an `m x n` matrix whose entry `(i, j)` is `d y_i / d x_j`. For training, that matrix is `1 x 100,000,000` — it *is* the gradient. The crucial fact is that **neither mode of automatic differentiation ever builds that matrix**. Each mode computes a *product* with it: - **Forward mode** computes `J u` for a chosen input-direction vector `u` — a **Jacobian-vector product** (JVP). - **Reverse mode** computes `v^T J` for a chosen output-direction vector `v` — a **vector-Jacobian product** (VJP). Everything about cost follows from those two shapes. ## Forward mode: one sweep per input direction Forward mode attaches a *tangent* to every value in the computation and updates it with the same rule as the value itself. If an intermediate is `z = g(x, y)`, then its tangent is `dz = (dg/dx) * dx + (dg/dy) * dy`, computed at the moment `z` is computed. You seed the input tangent with `u` and sweep forward once; at the end you hold the derivative of *every* output along `u`. Set `u = e_j`, the basis vector for input `j`, and one sweep returns **column `j`** of the Jacobian: how every output responds to that one input. To recover all `n` columns you need `n` sweeps. That count is not an implementation detail you can optimise away — there are `n` independent directions and one sweep resolves one of them. ## Reverse mode: one sweep per output direction Reverse mode runs the primal computation forward, then walks the graph backwards carrying *adjoints* (cotangents). The adjoint of a node is the derivative of the seeded output with respect to that node. At each operation the incoming adjoint is multiplied by that operation's local Jacobian **from the left** — which is exactly a vector-Jacobian product, and is why no local Jacobian is ever materialised either: the multiply is folded into a cheap closed-form rule. Seed the output adjoint with `v` and one sweep returns `v^T J`: the derivative of one output (or one fixed linear combination of outputs) with respect to **every** input. `m` sweeps recover all `m` rows. ## The arithmetic that decides training One scalar loss, 100 million parameters: - Reverse mode: `m = 1`, so **one sweep**. Its cost is a small constant multiple of one forward pass — roughly two — so a full gradient costs about three forward passes' worth of arithmetic, *regardless of how many parameters there are*. - Forward mode: `n = 100,000,000`, so **one hundred million sweeps**, each about the cost of a forward pass. If a forward pass takes 10 ms, that is about 1e6 seconds — roughly eleven days — for a single gradient, versus tens of milliseconds for the reverse sweep. Batching several tangent directions into one forward sweep amortises overhead but does not change the count: you still need `n` directions in total. ## Where 'backward costs about twice the forward' comes from Take a matrix product `Y = X W`, the operation most of a large network's arithmetic sits in. The forward pass is one matrix multiply. The reverse sweep needs two: `dX = dY W^T` to pass the adjoint further back, and `dW = X^T dY` to produce the parameter gradient — each with FLOP count comparable to the forward multiply. Summed over a matmul-dominated network, backward is about `2x` forward and a whole step about `3x`. It is a rule of thumb, not a law: elementwise operations have a cheap multiply as their backward rule and sit below the ratio, while operations whose backward rule re-derives quantities push it up. ## The asymmetric price Reverse mode is split into two phases, so intermediate values produced in the forward phase must survive until the backward phase consumes them; memory grows with the length of the computation. Forward mode produces value and tangent together and consumes both immediately, so it needs no such storage. The direction you pick trades sweeps against memory. ## The general rule The mode is decided by the *shape* of the function, not by whether it is a neural network. Many inputs and few outputs favours reverse mode; few inputs and many outputs favours forward mode. Training is the extreme case of the first shape, which is why reverse mode is the default everywhere in deep learning. ## What an interviewer is checking That you can say *why* one backward pass suffices — the counting rule, the scalar output, the constant-multiple cost — rather than reciting that backpropagation gives all the gradients at once.
- Where does the rule of thumb that the backward sweep costs about twice the forward pass come from?From matrix products. A forward `Y = X W` is one matmul; the reverse sweep needs two of comparable size, `dX = dY W^T` and `dW = X^T dY`. In a matmul-dominated network that gives roughly `2x` for backward and `3x` for the whole step. Elementwise operations fall below the ratio because their backward rule is a single cheap multiply.
- If the loss were vector-valued with 1,000 components, how many reverse sweeps would the full Jacobian take?One thousand — one per output row, each seeded with a different basis cotangent. At that point forward mode would be cheaper if the input count were below 1,000. In practice you usually do not need the full Jacobian; you need its product with a vector, which is one sweep in whichever direction matches.
- Does the cost of reverse mode depend on the number of parameters at all?The FLOPs of a single sweep grow with the size of the computation, which grows with parameter count. What does not grow is the *number* of sweeps: it stays at one. The claim worth stating precisely is that the gradient costs a constant multiple of one function evaluation, not that it is free.
Forward mode is measuring how the whole factory's output reacts when you nudge one machine — you must repeat it machine by machine. Reverse mode traces back from one defective product and attributes blame to every machine in a single walk.
saying these in an interview costs you the question
- Says the backward pass computes and stores the full Jacobian matrix
- Claims reverse mode needs one pass per parameter or per layer
- Thinks a single forward-mode sweep also returns every parameter's gradient
- Believes reverse mode costs no extra memory over the forward pass
- Cannot say what the seed value at the loss is or why it is 1