Why is a Monte-Carlo return unbiased while a one-step TD target is biased?

## The two targets Both methods update the same way — nudge the estimate toward a target — and differ only in the target. **Monte-Carlo:** wait until the episode ends, compute the realised discounted return from state `s`, ``` G = r1 + gamma*r2 + gamma^2*r3 + ... + gamma^(T-1)*rT V(s) <- V(s) + alpha * [G - V(s)] ``` **Temporal-difference:** after one step, ``` V(s) <- V(s) + alpha * [r + gamma * V(s') - V(s)] ``` ## Why Monte-Carlo is unbiased The true value of `s` under the current behaviour is *defined* as the expected return from `s`. The realised return `G` is one draw from that distribution. Averaging draws of a random variable converges on its expectation, so with first-visit Monte-Carlo — where each episode contributes at most one return per state — the estimate is unbiased for every sample size. A precise footnote worth knowing: **every-visit** Monte-Carlo, which averages returns from all visits to a state within an episode, is *biased* for finite data because returns from the same episode are correlated. It is still consistent, converging to the true value as data grows. Interviewers occasionally check whether you say "Monte-Carlo is unbiased" without qualification. ## Why Monte-Carlo has high variance `G` is the sum of everything that happened after `s`. Every stochastic action choice, every stochastic transition, every noisy reward between `s` and the terminal state contributes randomness. On a long episode, two visits to the same state can produce wildly different returns purely by luck downstream. That noise goes straight into the update, so estimates jitter and learning is slow — you need many episodes to average it out. ## Why TD has low variance The TD target depends on exactly two random things: the immediate reward `r` and the sampled next state `s'`. Everything beyond that is summarised by the deterministic lookup `V(s')`. One step of randomness instead of hundreds means dramatically less noise per update. ## Why TD is biased `V(s')` is an estimate, and early in training it is simply wrong — often initialised to zero. The target `r + gamma * V(s')` therefore does not have the true value as its expectation; its expectation is the true one-step-lookahead of a *wrong* value function. That systematic offset is the bias. The bias is **self-correcting rather than permanent**. As the estimates improve, the targets improve, and in the tabular case with a fixed policy and a suitable step-size schedule TD(0) converges to the true value function. The bias shrinks to zero; it is a transient cost, not a ceiling. It becomes a real hazard when the value function is approximated rather than tabulated, because then errors in `V(s')` can be structural — the representation simply cannot express the right value — and bootstrapping propagates that structural error to every predecessor state. ## Which one wins in practice Usually TD, and the reason is that variance dominates early learning. A high-variance unbiased target and a low-variance slightly-wrong target both point roughly in the right direction, but the noisy one requires far more samples to be trusted. Add TD's other advantages — updates after one step, no need to wait for episodes to end, works in continuing tasks — and TD is the default for prediction from streaming experience. Monte-Carlo earns its place when: - **the value estimates cannot be trusted** — poor initialisation, a bad state representation, or a function approximator whose errors are structural. Monte-Carlo never reads `V(s')`, so it cannot inherit its errors; - **the environment is not Markov** — the value of a state genuinely depends on history the state does not encode. TD's whole argument rests on the successor state summarising the future, so a non-Markov state breaks it; Monte-Carlo, which only sums actual reward, degrades far more gracefully; - **episodes are short**, in which case the variance penalty is small and the guarantee of unbiasedness is cheap. There is one more distinction worth carrying, because it explains their different failure modes. Given a fixed finite batch of experience, batch Monte-Carlo converges to the estimate with the smallest squared error on the observed returns, while batch TD(0) converges to the value function of the maximum-likelihood Markov model implied by that data — the *certainty-equivalence* estimate. TD is exploiting the Markov structure of the problem; Monte-Carlo is fitting the data it saw. That is exactly why TD wins when the Markov assumption holds and loses when it does not. ## Stating it correctly The crisp form is: Monte-Carlo targets are **unbiased, high variance**; TD targets are **biased, low variance**. Getting the direction backwards is a first-class error, and the reason is worth saying out loud — the bias comes from bootstrapping on an estimate, the variance from summing a whole trajectory of randomness.