Why must a REINFORCE run discard its collected trajectories after a single gradient update?

## Where the constraint comes from The policy gradient is `grad J(theta) = E over tau drawn from pi_theta [ sum_t G_t * grad log pi_theta(a_t | s_t) ]` Read the subscript on the expectation carefully. The distribution being averaged over is *the current policy's*. A Monte-Carlo estimate of an expectation is only valid if the samples come from the distribution named under the E. Your batch of trajectories came from whatever policy was in force when you collected it. The instant you apply an update, that is no longer the policy you hold, and reusing the batch silently estimates a different quantity: the gradient at the old parameters. For one tiny step the discrepancy is small, and in practice people do sometimes take two or three steps on a batch and accept the error. But it is an error, growing with how far the policy has moved, and unlike ordinary optimisation noise it does not average out: it is a systematic bias pointing back at where the policy used to be. ## Why it is worse than "the data is a bit stale" The drift is not only in which actions were chosen; it is in which **states were visited**. Consider a Gaussian policy pacing an hourly advertising budget, where the state includes how much budget remains for the rest of the flight. A policy that spends aggressively in the morning produces afternoon states with little budget left. Update the policy toward more conservative morning spending, and the afternoon states in your stored batch are states the new policy would rarely reach at all. You are computing a gradient over a state distribution that has become counterfactual. This compounding through the state visitation distribution is why off-policy correction in sequential decision problems is much harder than in a bandit or a supervised setting. ## The bill The accounting is unforgiving. Each gradient step consumes one batch of complete episodes, and each of those episodes is real interaction with the environment. In the ad-pacing case an episode is an entire campaign flight, so a training run that needs a few thousand updates needs a few thousand batches of flights. On a physical system, each episode also carries wall-clock time, wear, and the risk of a bad action being genuinely bad rather than merely uninformative. This is the structural difference from value-based learning. A Q-learning-style update is valid for transitions collected under any behaviour that covered the relevant actions, so a replay buffer can be revisited many times. A policy gradient batch has one shot. ## Partial escapes, and what each costs **Importance sampling.** You can reuse a batch collected under `pi_old` while holding `pi_new` by reweighting: multiply each term by the ratio `pi_new(a|s) / pi_old(a|s)`. In principle this is unbiased. In practice the ratios multiply along a trajectory and their variance grows quickly as the policies diverge; after a few updates a handful of trajectories with huge weights dominate the average and the estimate is worse than no reuse at all. The honest description is that importance weights extend the useful life of a batch from one update to a small number, provided you also keep the policy from moving far, and that constraining the step is itself a separate piece of machinery. **Bigger batches.** Collecting more episodes per update lowers the noise in each gradient step but does not reduce total interaction; if anything it raises it, because you spend more data to take the same number of steps. It buys stability, not economy. Choosing the batch size is a genuine trade between gradient quality and number of updates you can afford. **Parallel collection.** Running many copies of a simulator concurrently does not reduce the number of interactions required; it reduces the wall-clock time to gather them. That is the right fix when interaction is cheap but slow, and no help at all when interaction is intrinsically scarce or expensive. **Variance reduction.** Anything that lowers the variance of the estimator, a baseline being the cheapest, means fewer episodes are needed per usable gradient step. This is the highest-leverage lever available inside plain REINFORCE, because it attacks the number of samples needed rather than trying to recycle samples you already spent. ## What an interviewer is checking That you can name the exact reason: the expectation is indexed by the current policy. Weak answers say "the data is stale" or "on-policy means you learn online", both of which miss the mechanism. Strong answers connect it to the operating consequence, that gradient steps are metered in units of environment interaction, and to the honest limits of the workarounds: importance weights degrade with policy divergence, larger batches trade stability against update count, and parallelism buys clock time rather than data.