Where do you place stop-gradients in a model's graph, and what does each cut change?

## What the operation is A stop-gradient (also called a detach or a gradient barrier) is a node with two faces. **Forward:** it is the identity — the value passes through untouched, and the loss you compute is exactly the loss you would have computed without it. **Backward:** it returns zero to its input regardless of the incoming adjoint. Everything upstream of the cut therefore sees that value as a constant, as if it had been supplied by the dataset rather than produced by the model. A useful side effect is that the branch feeding the cut needs no stored intermediates for the backward pass. ## Cutting a path is not freezing a parameter These are routinely confused and they are different operations. - **Freezing** removes parameters from the update. The frozen layer still *transmits* gradient: a frozen block in the middle of a network happily passes gradient to everything before it. - **A stop-gradient** severs a *path*. The parameters on the other side of the cut are not frozen — if another live path reaches them, they still learn from that path. A shared trunk with one head's path cut still trains from the other heads. So the question "which parameters should stop moving?" is answered by freezing; the question "which route may the loss travel?" is answered by a stop-gradient. ## A decision framework For each edge you are considering cutting, ask four questions. 1. **Is this quantity conceptually a constant?** Weights, temperatures, thresholds and statistics that are *computed from* the model but intended as coefficients, not as things to be optimised, belong behind a cut. Their role in the loss is to modulate it, not to be tuned by it. 2. **Does the open path admit a degenerate solution?** This is the strongest reason to cut. If there is a way to lower the loss by manipulating the quantity instead of improving the prediction, gradient descent will find it — reliably and quickly. Cutting removes that route from the search space. 3. **Is the path's gradient meaningful?** Some routes carry gradient that is real but noisy, expensive to compute, or of the wrong sign for the behaviour you want. Cutting is then a bias-for-variance trade you should be able to state out loud. 4. **Can I write down what the objective becomes?** If you cannot say what function is now being minimised, the cut is a hack, not a design. ## The worked case: a self-paced curriculum Suppose each sample's contribution is scaled by a weight derived from the model's own per-sample loss — easy samples get more weight early, hard ones later. The weighted loss is `sum_i w_i * l_i`, and `w_i` is a function of `l_i`. Leave the path open, and `dL/dtheta` picks up a term through `w_i` as well as through `l_i`. Since making `w_i` small on a high-loss sample reduces the product immediately, the optimizer learns to *downweight what it cannot fit* instead of fitting it. The curriculum has silently become part of the objective, and the failure mode looks like plausible training: the reported weighted loss falls beautifully while unweighted validation loss stalls. Wrapping `w_i` in a stop-gradient restores the intended semantics — the weights are recomputed each step from the current model, but within a step they are constants. ## Asymmetry and role assignment Cutting one branch of an otherwise symmetric pair is how you give the two branches different jobs: one becomes the thing being fitted, the other a reference that only moves for reasons outside the loss. Whenever you rely on that, be explicit that the reference's update rule is now a separate design decision, because the loss no longer defines it. ## The cost of getting it wrong — in both directions **Too few cuts** and you get an objective with a shortcut in it: it trains, the loss curve is smooth, and the model solves the wrong problem. **Too many cuts** and you starve a branch that should have learned; it also trains, just worse, and you will blame the architecture. Neither failure shows up as an error, which is what makes this a judgment topic rather than a mechanics one. Because the loss curve will not tell you, test it directly: construct a loss, run one backward pass, and assert the *set* of parameters with nonzero gradient is exactly what you intended. That assertion is cheap, deterministic, and catches both a cut that migrated during a refactor and one that was never there. ## What to say in an interview Name the mechanism in one line, then talk about objectives rather than implementation: every stop-gradient is a claim about which quantities the optimizer is permitted to manipulate, and a model with undocumented cuts is a model whose objective nobody can write down.