skip to content

Why does backprop through time sum, not average, the per-step gradients of shared recurrent weights?

level: middleimportance: should knowfreq 46%

answer

  1. one parameter, many uses
  2. several paths to the same loss
  3. derivative along each path, then add
  4. any 1/T belongs in the loss, not the accumulation
  5. gradient norm tracks the unroll length

basics

~20 s

The same matrix is used at every time step, so the loss depends on it along T separate paths and its total derivative is the sum of those contributions. Averaging would give the gradient of a rescaled objective instead.

solid answer

~50 s

Unrolling makes one hidden-to-hidden matrix `W` appear at T places in the graph, so the loss depends on `W` through T routes and `dL/dW` is the sum of the per-route terms: `dL/dW = sum_t delta_t h_(t-1)^T`, one term per step, where `delta_t = dL/dh_t`. Nothing in the derivative introduces a `1/T`; dividing by T would be the exact gradient of the loss `L/T`, which is a different objective. That is still a legitimate choice — many people define the sequence loss as the *mean* of per-step losses so gradient scale does not depend on sequence length — but then the `1/T` sits in the loss definition, and the accumulation over shared weights is still a plain sum. The practical consequence is that under a summed loss the gradient norm grows with the unroll length, so a learning rate tuned at 20 steps can be far too large at 200.

code

python · 24 lines
python
xs = [1.0, 2.0, 3.0]          # one scalar input per time step
w = 0.5                       # the shared hidden-to-hidden weight

hs = [0.0]                    # h_0 = 0
for x in xs:                  # forward: h_t = w * h_(t-1) + x_t
    hs.append(w * hs[-1] + x)

g = 1.0                       # dL/dh_T for the loss L = h_T
grad_w = 0.0
for t in range(len(xs), 0, -1):
    grad_w += g * hs[t - 1]   # step t's own contribution to the shared w
    g *= w                    # carry dL/dh back one step

def forward(w):               # closed-form check by finite difference
    h = 0.0
    for x in xs:
        h = w * h + x
    return h

fd = (forward(0.5 + 1e-6) - forward(0.5 - 1e-6)) / 2e-6

print(hs)             # [0.0, 1.0, 2.5, 4.25]
print(grad_w)         # 3.0  -- the SUM of the three per-step contributions
print(round(fd, 6))   # 3.0  -- same number, so summing is the true derivative

go deeper

for a junior

Recall that a weight reused at every step collects a contribution from every step, and that those contributions are added together before a single update is applied.

for a middle

Be able to write dL/dW = sum_t delta_t h_(t-1)^T and to explain that a 1/T is a choice made in the loss definition, never something the derivative produces on its own.

for a senior

Diagnose the symptom: a model that trained cleanly at a short unroll and diverges at a long one, under a summed loss, is usually a gradient-scale problem, not an architecture problem.

for a principal

Standardize the reduction convention across the team's experiments and require it to be reported with learning rates, so results at different sequence lengths remain comparable.

## The setup Take the simplest recurrent update, ignoring the nonlinearity for a moment: `h_t = W h_(t-1) + U x_t` `W` is the hidden-to-hidden matrix. It is a single object in memory, and the forward pass uses it at step 1, at step 2, ... and at step T. When the graph is unrolled for backpropagation, that one matrix appears at T places in the graph. ## The rule The loss L is a function of `W` through several distinct routes: through the multiplication at step 1, through the multiplication at step 2, and so on. When one quantity influences an output along several routes, the derivative of the output with respect to that quantity is the **sum** of the derivatives along each route. So: `dL/dW = sum over t of (dL/dh_t at step t) * (partial h_t / partial W)` and with `partial h_t / partial W = h_(t-1)` (as an outer product in the matrix case), this reads `dL/dW = sum_t delta_t h_(t-1)^T`, where `delta_t = dL/dh_t`. Each time step contributes exactly one term. There are T terms. They are added. Nowhere does a factor of `1/T` appear, because nothing in the derivative asked for an average. ## Why averaging is wrong (and what it actually does) Dividing by T does not produce a "more stable version of the same gradient" — it produces the exact gradient of a *different* loss, namely `L/T`. If you want that loss, define it that way. That is a real and legitimate choice: many practitioners define the sequence loss as the **mean** of per-step losses rather than their sum, precisely so that the gradient's magnitude does not depend on how many steps happen to be in the batch. But the `1/T` then lives in the loss definition, and the accumulation rule over the shared weights is still a plain sum. Confusing the two is what produces the mystery of "my model trained fine at 20 steps and diverged at 200". ## The practical consequences **Gradient magnitude tracks unroll length under a summed loss.** With per-step losses summed over time, roughly T times as many contributions land on `W`, and the gradient norm grows with T. Change the unroll length and your effective step size changes with it. Keep the reduction (sum vs mean over steps) fixed when you compare experiments, and state which one you used. **Batch reduction and time reduction are different axes.** Almost everyone averages over the batch dimension so the gradient does not depend on batch size. Far fewer people notice that time is a second axis with the same question attached. "Mean over batch, sum over time" and "mean over both" are different objectives and want different learning rates. **Adaptive optimizers hide it, imperfectly.** An update rule that divides by a running estimate of the gradient's own scale is largely insensitive to a constant rescaling of the loss, so switching sum to mean over time changes less than it would under plain gradient descent. "Largely" is not "exactly": the epsilon in the denominator and any decoupled weight decay do not rescale with the gradient, so the correspondence is approximate. **Every reused parameter behaves this way, not just `W`.** The input-to-hidden matrix `U` and the bias `b` are also used once per step and also collect one contribution per step. What is special about `W` is not that it is summed — it is that `W` additionally sits on the path *between* steps, so it is the matrix through which gradient must repeatedly travel to reach earlier steps. ## Tracing it by hand Scalar version: `h_t = w * h_(t-1) + x_t`, three steps, `h_0 = 0`, inputs 1, 2, 3, `w = 0.5`, loss `L = h_3`. Forward: `h_1 = 1.0`, `h_2 = 2.5`, `h_3 = 4.25`. Expanding, `h_3 = w^2 * x_1 + w * x_2 + x_3`, so `dL/dw = 2w*x_1 + x_2 = 1.0 + 2.0 = 3.0`. Running the backward sweep instead: start with `dL/dh_3 = 1`; step 3 contributes `1 * h_2 = 2.5`; carry the gradient back by multiplying by `w`, giving 0.5; step 2 contributes `0.5 * h_1 = 0.5`; carry back again to 0.25; step 1 contributes `0.25 * h_0 = 0`. Total: `2.5 + 0.5 + 0 = 3.0`. The sum of per-step contributions is the derivative; the average, 1.0, is not the derivative of anything you asked to minimize. ## What a strong answer sounds like "The unrolled graph uses the same `W` at every step, so `W` influences the loss along T paths and its gradient is the sum of the per-step terms — one `delta_t h_(t-1)^T` per step. If I want length-invariant gradient scale I put the `1/T` in the loss by averaging the per-step losses, not in the accumulation, and I say which convention I used when I report a learning rate."

  • If I switch the per-step loss reduction from a sum over time to a mean, what actually changes?
    The loss, and therefore the gradient, shrinks by a factor of T. Under plain gradient descent that is identical to scaling the learning rate by `1/T`. Under an update rule that divides by a running estimate of the gradient's own magnitude, the rescaling largely cancels — but only largely, since a stabilizing epsilon and decoupled weight decay do not rescale with the gradient.
  • Does the input-to-hidden matrix get the same summed treatment as the hidden-to-hidden one?
    Yes. It is also used once per step, so it also collects one contribution per step and its gradient is a sum over time. What is special about the hidden-to-hidden matrix is not the summation but its position: it sits on the path connecting consecutive steps, so gradient travelling back to earlier steps must repeatedly pass through it.
  • Why is 'mean over the batch, sum over time' such a common source of confusion?
    Because almost everyone remembers to normalize the batch axis and forgets that time is a second axis with the same question attached. Two teams can run the same architecture, the same learning rate and the same data with different time reductions and see wildly different stability. Report which convention you used alongside any learning rate.

saying these in an interview costs you the question

  • Says each time step performs its own separate weight update
  • Averages over time steps to keep the gradient small, calling it the same gradient
  • Thinks weight sharing means only the last step's gradient counts
  • Confuses averaging over the batch with averaging over time
  • Cannot say where a legitimate 1/T factor belongs

context

resolving links…