How do you choose the truncation window length in truncated backprop through time?

## What truncated BPTT actually does Full backpropagation through time unrolls the entire sequence and backpropagates from the last step to the first. That is fine for a 30-token sentence. It is impossible for a stream with hundreds of thousands of steps, and merely expensive for a few thousand. Truncated BPTT cuts the stream into consecutive windows of k steps and trains window by window: 1. Run the forward pass over the next k steps, starting from the hidden state left behind by the previous window. 2. Compute the loss on those k steps and backpropagate **only within the window** — the gradient stops at the window's first step and does not cross into the previous window. 3. Update, then carry the final hidden state forward as the starting state of the next window, but carry it as a *value*, with no gradient attached to it. The two halves of step 3 are what candidates most often get wrong. The forward context is **not** reset: the hidden state entering window n+1 still summarizes everything the network saw in windows 1 through n, so the model's prediction at step 5,000 can in principle depend on step 3. What is reset is the gradient path: no derivative connects a parameter update to anything more than k steps back. ## What the window length buys you **Gradient reach.** k is the horizon over which the network can *learn* to route information. If the label depends on something that happened 150 steps earlier, no gradient path of length 20 ever tells the weights to preserve it, so the dependency is not learnable from the truncated objective — the model may pick up a weaker correlate, but not the mechanism. If your click-stream session model has to connect a purchase to a filter the user applied 150 events ago, a window of 20 will not do it and a window of 200 will; that decision, not the memory budget, is the primary input. **Cost, linearly.** Doubling k roughly doubles the backward work per window and roughly doubles the per-step forward values retained across the window. Since a window covers twice as many steps, the cost *per training step of sequence* is roughly flat — what grows is the size of one unrolled chunk, the latency of one update, and how few updates you get per pass over the data. Longer windows also mean fewer, larger, more correlated updates. **Bias.** The truncated gradient is not an unbiased estimate of the full-sequence gradient. It is the exact gradient of a different (truncated) objective: every path longer than k has been dropped, not sampled away. Increasing k shrinks the bias; it never randomizes it into noise. A published variant reduces it by decoupling the two lengths — backpropagate over k2 steps every k1 forward steps with k2 > k1, so windows overlap and some cross-boundary paths are recovered — at the cost of redundant backward work. ## How to actually choose k 1. **Start from the dependency horizon, measured in steps.** Ask what the longest range dependency you need is, in the units the model sees. This is a data question, and it is often answerable: how far back is the referent, how long is a session, how many samples separate cause and effect. 2. **Set k above that horizon if you can afford it.** Then check affordability: one window's backward pass and retained forward values scale with k, and one update now covers k steps. 3. **If you cannot afford it, shorten the sequence rather than the horizon.** An hour of a signal sampled at 250 Hz is 900,000 steps; no window covers a dependency spanning minutes at that rate. Downsample, aggregate into coarser units, or summarize sub-segments so that the same real-world dependency spans hundreds of steps instead of hundreds of thousands. Changing what a "step" means is usually cheaper than fighting the window length. 4. **Validate empirically.** Sweep k over a small grid and watch a metric that genuinely requires long context, not just aggregate loss — aggregate loss is dominated by short-range structure and will barely move while the long-range behavior you care about is still broken. 5. **Keep windows contiguous and in order.** Carrying state across windows only makes sense if window n+1 really follows window n in the same stream. Shuffling windows destroys the carried context, and mixing series in one slot corrupts it outright. ## Common traps Resetting the hidden state at each window (throws away the forward context for no reason, though it is the right thing at a genuine sequence boundary). Assuming a bigger window is strictly better — it costs proportionally and gives you fewer updates. Assuming the carried state substitutes for gradient reach — gating helps the forward state *hold* information, but no amount of gating lets a gradient cross a truncation boundary. And treating the truncated gradient as unbiased when diagnosing a model that will not learn a long-range rule. ## The one-line version k is the length of the gradient's memory. Choose it from the dependency you must learn, pay for it linearly, and if the number is unaffordable, redefine the step so the dependency gets shorter.