Value Networks

## What a replay buffer is A replay buffer (or experience replay memory) is a fixed-capacity store of transitions. Each entry is a tuple `(s, a, r, s', done)`: the state the agent was in, the action it took, the scalar reward it received, the next state, and whether the episode ended. The agent runs in the environment and appends each transition it experiences; when the buffer is full, the oldest entry is evicted (first-in, first-out). Training does not read the stream directly. Instead, every gradient step draws a minibatch — commonly a few dozen transitions — uniformly at random from the buffer, and fits the value network on that batch. ## Problem one: consecutive samples are correlated Stochastic gradient methods are built on the assumption that each minibatch is a roughly independent sample from the distribution you want to do well on. The minibatch gradient is then an unbiased, reasonably low-variance estimate of the true gradient, and the noise across steps partially cancels. A raw environment stream breaks that assumption in the strongest possible way. Consider an agent playing a paddle-and-ball game from stacked frames: four consecutive time steps of one rally show the ball a few pixels apart, the paddle in nearly the same place, the same action repeated, and usually the same reward of zero. A batch of those four is not four examples; it is closer to one example counted four times. Worse, the correlation is not just redundancy — it is *directional*. During a long stretch of one behaviour the batches all point the same way, so the network overfits that stretch, and then the next stretch drags it back. The result is a value function that oscillates with whatever the agent happens to be doing, rather than converging. Uniform sampling from a large buffer fixes this structurally. A single batch drawn from a million stored transitions contains states from many different episodes, many different game situations and many different past policies. Successive batches overlap only by chance, so the per-step gradient noise looks much more like the independent noise the optimiser expects. ## Problem two: data thrown away after one look Environment interaction is usually the expensive part: stepping a simulator, or worse, acting in a real system. Learning online from each transition once uses every collected sample exactly once. With a buffer, a transition remains available until it is evicted, so it contributes to many gradient steps. There is a simple piece of arithmetic worth remembering: if you insert one transition and draw one batch of size B per environment step, then a transition that survives the full capacity of the buffer is expected to be drawn about B times before eviction — the capacity changes how *old* the average sample is, not how many times it is reused. ## Why this only works off-policy A transition in the buffer was generated by whatever policy the agent had at the time. The Q-learning target uses `r + gamma * max_a' Q(s', a')` — the maximum over next actions, not the action the behaving policy would have chosen — so it does not care which policy produced the data, as long as the environment's dynamics and rewards are unchanged. That is exactly what makes replay legitimate here. An on-policy method whose update is defined in terms of the action the *current* policy takes cannot simply reuse a buffer of old behaviour without correction; naively bolting replay onto such a method is a real error, not a shortcut. ## What replay does not give you A few honest limits. First, samples from a buffer are not independent and identically distributed in any strict sense — they are drawn from a non-stationary mixture of every policy the agent has had, and consecutive draws can still be from the same episode. Replay reduces correlation; it does not eliminate it. Second, the loss is now averaged over the buffer's state distribution rather than the current policy's, which is a genuine change in what is being optimised: the network spends capacity on regions the agent may no longer visit. Third, replay does nothing about the fact that the regression target is itself produced by the network being trained — that instability needs a separate mechanism. ## Practical shape A buffer is typically a preallocated ring of arrays rather than a list of objects, because the dominant cost is memory: storing raw stacked image frames for a million transitions is large enough that implementations share frames between overlapping stacks and store them at low precision. Learning normally does not begin until the buffer holds some minimum number of transitions, so the first batches are not drawn from a handful of near-identical starting states.