Why does a replay-based deep RL learner need far fewer environment steps than a PPO-family run?

## What sample efficiency means here In reinforcement learning, *sample efficiency* is measured in **environment steps**: how many transitions — a tuple of state, action, reward and next state — the agent has to actually collect before it first reaches some chosen score. RL learning curves are plotted with environment steps consumed on the x-axis, not epochs and not wall-clock hours, precisely so that two algorithms with wildly different compute costs can be compared on the currency that is usually the scarce one: interaction with the environment. The more sample-efficient method is the one whose curve crosses the threshold further to the left. A compact way to report it is *steps-to-threshold*: the number of environment steps consumed before evaluation returns first reach, say, a fixed score on the benchmark. Reporting a final score alone hides the whole question, because a weak method usually catches up if you let it interact for long enough. ## The two families and their reuse counts **On-policy policy-gradient methods of the PPO family** collect a fresh batch with the current policy — a common size is a couple of thousand transitions gathered across parallel actors — perform on the order of ten epochs of minibatch updates over that batch, and then discard it. Their gradient estimator is defined with respect to the distribution induced by the policy that is being improved; once the parameters have moved, the batch is no longer a sample from the right distribution, and clipping or trust regions only stretch that window a few epochs wide. **Replay-based off-policy learners** — the DQN family for discrete actions, and off-policy actor-critics such as TD3 and SAC for continuous control — write every transition into a buffer holding on the order of a million slots and train by sampling uniform (or prioritised) minibatches from it. Old transitions stay usable because the learning signal is a bootstrapped Bellman target: it asks what reward and next state followed this action in this state, which is a fact about the environment's dynamics, not about which policy happened to choose the action. ## The arithmetic that produces the gap Do the counting explicitly, because that is what the question is really testing. - On-policy: a 2048-transition batch used for ten epochs means each transition contributes to about **ten** gradient computations, then it is gone forever. Over a run of N environment steps, the total number of sample-uses is about 10N. - Off-policy: one gradient step on a minibatch of 256 taken after every environment step means 256 sample-draws per collected step. Over a one-million-step run that is 256 million draws spread over a buffer that holds at most one million distinct transitions — roughly **two hundred** uses per transition on average. So the replay learner extracts one to two orders of magnitude more learning from each step it pays for. That is the mechanical reason a replay-based value learner and a PPO-family run that end up at the same score on the same locomotion benchmark can differ by roughly an order of magnitude in the interaction budget they needed to get there. ## What the reuse costs The efficiency is not free, and a good answer says so: - **Compute per environment step is far higher.** The replay learner runs a full gradient update (often several) for every single step of interaction; the on-policy run amortises ten epochs across thousands of steps. If the simulator is cheap and parallelisable, the on-policy method can finish sooner in wall-clock hours while consuming ten times the steps. - **Off-policy learning is more delicate.** Bootstrapping from a learned value function that is itself trained on its own targets brings overestimation and divergence risks that on-policy methods, which regress against actually observed returns, largely sidestep. - **The gap is task-dependent.** It is widest on continuous-control tasks with dense rewards, where value learning is well conditioned. It narrows where the value function is hard to fit or the reward is very sparse. ## The distinction to keep straight Sample efficiency and throughput are different currencies. An agent that needs fewer environment steps but more GPU-hours is more sample-efficient and less compute-efficient. Which one you should optimise is decided by which resource is actually scarce: a slow, licensed, or physical environment makes steps precious; a fast vectorised simulator makes them nearly free and turns the question into one about wall-clock time. ## A common wrong answer Candidates often say the replay learner is simply a better algorithm, or that PPO is inefficient because it is conservative. Neither is the mechanism. The clipping in a PPO-family objective limits how far the policy moves per update; it does not decide how many times a transition is reused. The reuse count does, and that is what you should quantify when asked.