What separates model-free from model-based reinforcement learning?

## The object in question: the model In reinforcement learning an agent sits in a state, takes an action, receives a reward and lands in a new state. A **model** of the environment is a description of that last part: the transition dynamics `p(s' | s, a)` (where you end up) and the reward function `r(s, a)` (what you get). Having a model means you can ask "what would happen if…" **without acting** — you can simulate. The model-free / model-based split is simply: does the agent hold such a description, or not? ## Model-based learning A model-based agent either - **is given** the dynamics (a board game's rules, a queueing system written down as equations, a physics simulator), or - **learns** them from data — count how often action `a` in state `s` led to `s'`, average the rewards, and you have an estimated transition and reward model. With a model in hand the agent can compute a policy by reasoning over it, and it can generate as much synthetic experience as it likes. That is why model-based methods tend to be far more **data-efficient**: one observed transition improves the model, and the improved model can be replayed thousands of times. The costs are real. The model has to be accurate, and errors compound: an agent that plans many steps ahead through a slightly wrong model will confidently chase rewards that do not exist — "model exploitation". Learning the dynamics of a rich environment can also be much harder than learning the value of being in a state, because the dynamics carry detail the decision does not need. ## Model-free learning A model-free agent never represents `p(s' | s, a)`. It keeps only what it needs to act — a value function, a policy, or both — and updates it from experience it actually had. Two families dominate: - **Monte-Carlo**: wait until the episode ends, compute the realised return `G` from a state, and move that state's estimate toward `G`. - **Temporal-difference**: after a single step, move the estimate toward `r + gamma * V(s')`, the reward you just got plus your current estimate of where you landed. Both learn the *consequences* of behaviour without ever learning the *mechanics* of the environment. Nothing in a TD update requires knowing why the conveyor jams; it only requires having seen it jam. ## Which one the situation forces The deciding questions are practical: 1. **Can the dynamics be written down or reliably estimated?** For the conveyor, jams depend on part geometry, humidity, wear and operator behaviour. Nobody can enumerate that state space honestly, and a badly estimated model is worse than none. Model-free. 2. **Is interaction cheap?** If every sample costs a production run, the data efficiency of a model becomes very attractive — even an imperfect learned model may pay for itself. 3. **Do you need to answer counterfactuals?** "What if we changed the reward for early completion?" is a model question. A model-free value function is tied to the behaviour it was trained under. ## Why this matters for TD specifically Temporal-difference learning is the canonical model-free prediction method, and it borrows one property from each side of the divide. Like Monte-Carlo, it learns purely from sampled transitions and needs no model. Like model-based reasoning, it **bootstraps** — it updates an estimate using another estimate, which is exactly the kind of self-consistency a model-based computation enforces. That hybrid character is why TD is often described as sampling the experience but reasoning over the estimates, and it is the reason TD can update after one step rather than one episode. A final clarification, because interviewers probe it: *model-free* does not mean *no function approximator*. A model-free agent may hold a large parametric value function. "Model" here means a model of the **environment's dynamics**, not the statistical model doing the estimating.