Planning and Learning

Policy gradient methods tune the policy's parameters directly, so an action is sampled from a distribution. Value-based control has to maximise a value estimate over all actions at every step, which is intractable when the action is a real-valued vector.

Policy iteration repeats two steps. Evaluation sweeps every state, replacing its value with the expected return of the current policy. Improvement then rebuilds the policy greedily from those values; when no action changes, the policy is optimal.

REINFORCE weights the gradient of each action's log-probability by the return that followed it: theta <- theta + lr * G_t * grad log pi(a_t | s_t). Actions from high-return episodes get more probable. The reward itself is never differentiated.

Q-learning's target uses the maximum action value in the next state, so it learns the greedy policy while behaving some other way, which makes it off-policy. SARSA's target uses the action actually taken next, so it evaluates the exploring policy itself: on-policy.

After one transition, TD(0) moves the estimate a fraction of the way toward the observed reward plus the discounted estimate of the next state: V(s) <- V(s) + alpha * [r + gamma * V(s') - V(s)]. The bracketed quantity is the TD error.

Reinforcement learning earns its cost when the only outcome you can measure arrives after a long sequence of decisions, so no per-decision target exists. If you can write an honest label for each decision, supervised learning is cheaper and safer.

A model-based agent has (or learns) the environment's transition and reward dynamics and can plan against them. A model-free agent never builds those dynamics; it estimates values or a policy directly from sampled experience. Temporal-difference learning is model-free.

Policy iteration evaluates the current policy to convergence, then makes it greedy, repeats. Value iteration folds both into one sweep that takes the maximum over actions. Both reach the same optimal values, trading cost per round against number of rounds.

Weighting by return minus a baseline centres the weights, so better-than-average actions are pushed up and worse-than-average ones down, instead of everything rising. Any baseline independent of the chosen action leaves the gradient unbiased.

A first-visit Monte-Carlo return is an actual sampled return, so on average it equals the true value — unbiased, but noisy because it accumulates every random reward to the end of the episode. A TD target uses the current estimate of the next state, which is wrong early on, so it is biased but far lower variance.

Trial-and-error learning needs enormous numbers of episodes, so reinforcement learning is feasible only where one episode is cheap, fast and safe. A simulator can yield a million overnight; an agent whose episode is a fiscal quarter yields four a year.

In every state the greedy action is worth at least as much as the current policy's action, and that advantage applies again at the next step. Unrolling it shows the greedy policy's return dominates everywhere — the policy improvement theorem.

The policy gradient is an expectation over trajectories drawn from the current policy. One update moves the policy, so the batch now comes from a different distribution and estimates the wrong expectation. Every step needs fresh interaction.

Q-learning bootstraps from the greedy next action, so its values describe an agent that never steps off the cliff, and the short edge path looks best. SARSA bootstraps from the exploratory action actually taken, so occasional random falls depress the edge cells' values.

Trained only on logged decisions, an offline policy extrapolates the value of doses clinicians rarely gave, and maximisation selects exactly those over-optimistic estimates. You also cannot test it without acting, and the unrecorded reasons behind each dose confound the data.

The policy is exactly optimal for the model you supplied, not for the world. Errors in the transition probabilities compound over the discounted horizon, and the planner actively exploits any transition whose probability or reward was estimated too optimistically.

Pick n on the delay between action and reward. Small n gives low-variance but myopic targets that propagate credit one state per pass; large n propagates credit fast but carries the noise of a long trajectory. Tune n empirically; intermediate values usually beat both extremes.

Prefer a constant learning rate when the environment drifts, because it keeps weighting recent experience and lets the agent track change. Prefer a decaying one when the problem is stationary and you want the convergence guarantee, which a constant rate forfeits.

Build a simulator when the dynamics are well understood and the real system is too slow or unsafe to explore. It is a trap when the hardest part to model is the part that matters: the agent optimises your assumptions.