In REINFORCE, why does subtracting a baseline from the return cut gradient variance?

## The problem with raw returns as weights The REINFORCE estimator weights each step's score `grad log pi_theta(a_t | s_t)` by the return `G_t`. Consider a portfolio-rebalancing agent evaluated on episode profit, where returns are large, always positive, and swing wildly between episodes: one week returns 40, the next 900, the next 120. Every one of those weights is positive. So every sampled action, good or bad, gets its probability pushed *up*. The only thing distinguishing a good action from a bad one is that its nudge is somewhat larger, and that difference is buried under the enormous variation in the overall scale of returns. The estimator still points uphill in expectation, but a single batch's estimate is dominated by which episodes happened to land in a rich period, not by which actions were better than the alternatives. That is variance in the pure statistical sense: the estimator is right on average and useless per sample. ## The identity that makes baselines free Subtract a constant `b` from every weight: `grad_estimate = sum over t of (G_t - b) * grad log pi_theta(a_t | s_t)` The bias introduced is `-b * E[ grad log pi_theta(a | s) ]`. That expectation is exactly zero, and the proof is two lines: `E over a from pi[ grad log pi(a|s) ] = sum over a of pi(a|s) * grad log pi(a|s) = sum over a of grad pi(a|s) = grad ( sum over a of pi(a|s) ) = grad 1 = 0` The probabilities have to sum to one no matter what `theta` is, so the gradient of that sum is zero. This is the **expected score is zero** identity, and it is the single most useful fact about score-function estimators. It means you may subtract any quantity that is constant with respect to the action being averaged over, and the estimator's expectation is untouched. Only the variance changes. ## What the baseline changes about the signal With `b` set to roughly the mean return, the weight `G_t - b` becomes a comparison: how much better than typical was this episode? Above-average episodes get positive weights and their actions become more likely; below-average episodes get negative weights and their actions become less likely. The estimator now carries information about *relative* quality, which is what actually determines the optimal policy, instead of absolute scale, which does not. Shifting every reward in a task by +1000 changes nothing about the optimal policy, and with a baseline it changes nothing about the estimator either. Without one it inflates the variance enormously. The cheapest useful baseline is a **running average of recent episode returns**, updated as training proceeds. It costs one scalar, requires no extra model, and captures most of the benefit. A more refined choice makes the baseline depend on the state, since some states are simply richer than others; a per-timestep average handles tasks where early and late parts of an episode pay very differently. ## What a baseline may and may not depend on The validity condition is precise: the baseline must not depend on the **action** whose log-probability is being differentiated. It may depend on the state, on the timestep, on the entire training history, on a running statistic of past episodes. All of those are constants with respect to the action expectation, so they factor out of the sum and hit the zero identity. Make `b` depend on the sampled action and it no longer factors out. The `-b(s,a) * grad log pi(a|s)` term keeps a non-zero expectation, and the estimator is biased: it will converge to the optimum of some other objective. This is the single most common way people break a baseline in practice, usually by accident, for example by computing the baseline from the same sample's outcome rather than from a statistic that was fixed before the action was drawn. ## Is the mean return the best baseline? Not exactly. The variance-minimising baseline is a weighted average of returns, weighted by the squared magnitude of the score, rather than the plain mean. In practice nobody bothers: the plain running mean captures the bulk of the reduction, and the optimal baseline requires estimating second moments that are themselves noisy. The distinction is worth knowing because it shows the baseline is a variance-reduction device with an optimum, not a normalisation ritual. ## Common confusions worth clearing - **A baseline is not a reward change.** The rewards, the objective, and the optimal policy are all identical with and without it. Only the noise of the estimator differs. - **A baseline is not the same as scaling.** Dividing the weights by their standard deviation is a separate trick; it changes the effective step size and is not covered by the zero-expectation argument in the same way. - **A baseline does not eliminate the need for samples.** It reduces variance per sample; it does not make a single episode informative. The practical summary: it costs one running scalar, keeps the estimator unbiased by an identity that holds exactly, and often shrinks variance by an order of magnitude. There is essentially no reason to run REINFORCE without one.