Recurrence and BPTT

## What "unrolling" means A recurrent network is written as a loop. At each time step `t` it takes the input `x_t` and the previous hidden state `h_(t-1)` and produces a new hidden state: `h_t = f(W h_(t-1) + U x_t + b)` where `f` is an elementwise nonlinearity such as tanh, `W` is the hidden-to-hidden matrix, `U` the input-to-hidden matrix, `b` the bias, and `h_0` is a zero (or learned) starting state. Written this way the graph has a cycle: `h` feeds back into itself. Gradient-based training needs a graph without cycles, because a derivative has to be taken along a definite path from parameters to loss. Backpropagation through time (BPTT) removes the cycle by *unrolling*. For a concrete sequence of length T, the loop is expanded into T copies of the cell laid out in a chain: copy 1 consumes `h_0` and `x_1` and emits `h_1`; copy 2 consumes `h_1` and `x_2` and emits `h_2`; and so on to copy T. Nothing feeds backwards, so the result is an ordinary feed-forward network that happens to be T layers deep — deep in *time* rather than in stacked layers. Backpropagation then applies to it exactly as it applies to any other feed-forward network. ## What the copies share, and what they do not The copies share **parameters**. There is one `W`, one `U`, one `b`, and every copy uses those same tensors. Unrolling does not create T layers with T sets of weights; it creates T *uses* of one set of weights. This is the single most common misunderstanding of the picture, and it is what makes the backward pass interesting: a parameter that is used T times receives a gradient contribution from each of those uses. The copies do **not** share activations. Each step has its own `x_t`, its own `h_t`, and its own intermediate pre-activation values. Those per-step values are what the backward pass needs in order to evaluate the local derivatives, so the forward pass has to keep them around until the backward pass consumes them. ## Where the loss enters Two shapes are common, and they change where gradient is injected: - **Sequence-to-label.** One loss is computed from the final hidden state (or a pooled summary). The gradient enters at step T only, and travels backwards from there through T-1, T-2, ... to step 1. - **Per-step output.** A loss is computed at every step, for example next-token prediction over a paragraph of text or a tag per token. Every step injects its own gradient at its own output, and the total loss is a sum (or mean) over steps. Step t's parameters then accumulate gradient from step t's own loss *and* from every later loss whose path runs back through `h_t`. In both shapes the backward pass runs from the last step to the first, because the gradient at step t depends on the gradient already computed at step t+1. That strict ordering is why the backward pass over time is inherently sequential. ## What it costs Both the forward and the backward pass do work proportional to T, and the forward values that must be retained for the backward pass also grow with T. Doubling the number of unrolled steps roughly doubles the backward work and roughly doubles the retained per-step values. For a 30-step sentence that is nothing. For an hour of a physiological signal sampled at 250 Hz — 900,000 steps in one "sequence" — a full unroll is not something you can build at all, and training instead processes the stream in windows with the gradient cut at the window boundary. Note that sequence length also has to be handled per batch: sequences in the wild are not all the same length, so the unrolled depth differs from batch to batch. Nothing about BPTT requires a fixed T; the unrolled chain is simply built to whatever length the current batch needs. ## Why interviewers start here Almost every hard fact about recurrent training falls out of this picture. The gradient of the shared weights is a sum over steps because the weights are used once per step. Long-range learning is hard because the gradient must survive a trip back through many steps. Truncation exists because the unrolled chain's cost is linear in its length. Gating architectures exist because of what happens along that long backward path. A candidate who can draw the unrolled chain and point at where the weights are shared and where the loss enters can reason their way to all of it; a candidate who cannot will guess. ## A compact summary Unroll the loop into T copies of one cell; attach inputs and losses; run backpropagation over the resulting acyclic graph from step T down to step 1; every copy contributes gradient to the same shared parameters; the cost of all of this is linear in how many steps you unrolled.