When does a delayed payoff justify reinforcement learning over supervised next-step prediction?

## What supervised learning needs, and what it sometimes cannot get A supervised model learns a mapping from inputs to targets, and it needs one target per training example. For a decision problem that means a concrete statement of the form: *given this situation, this was the action that should have been taken*, or at least *this action led to this measurable consequence*. Sometimes that target genuinely exists. An expert may have recorded the correct decision, or the consequence may be immediate and cleanly attributable, so each decision can be labelled by its own result. Reinforcement learning is for the case where no such target exists and none can be manufactured honestly. What you can observe is an outcome -- often a single number, often much later -- produced by a whole sequence of decisions acting together. Nobody can tell you which of the fifty decisions along the way deserved the credit. ## Delayed credit assignment is what RL actually sells Consider a tutoring agent that picks which exercises a student works on in each weekly session across a term, and is judged on the end-of-term exam score. There is no label saying which exercise was the right choice in week three. There is one score, in week twelve. *Credit assignment* is the problem of attributing that late score back to the earlier decisions. RL's core machinery exists to do this: a **return** (the accumulated outcome from a point in the sequence onward) and a **value estimate** (the expected return from a situation, or from a situation-action pair) convert one delayed number into a per-decision learning signal. That conversion, not the vague idea of "trial and error", is the capability you are buying, and it is what makes the extra cost of RL rational. ## The myopic-proxy trap The standard alternative is to invent a per-decision label out of something measurable right now: did the student get the next exercise right, did the shift meet today's quota, did the outcome improve within the hour. This is very often the correct engineering answer. It trains on ordinary supervised infrastructure, it is far easier to evaluate offline, and it fails in ways people already know how to debug. It goes wrong when the proxy can be maximised by behaviour that damages the real objective. A tutor optimised for next-exercise correctness will serve easy exercises: immediate accuracy climbs and exam scores fall. The proxy and the goal agree in the historical data and diverge exactly in the region an optimiser will push into. So the diagnostic is not "is my proxy correlated with the outcome?" but "does the correlation survive being optimised hard?" ## Delayed label is not the same as delayed credit This distinction separates the people who have thought about it from the people who have not. A subscription cancellation that lands three weeks after a pricing decision is a *delayed label*: it is late, but it still attaches to one identifiable decision. That is a slow supervised problem -- you wait, you join, you train, you accept the staleness. Delayed *credit* is different: the outcome cannot be attributed to any single decision because it was produced jointly by many of them. Only the second case needs RL's machinery. ## A checklist for the framing decision 1. **Write down the label.** Literally state what the supervised target for one decision would be. If you cannot state it, you are on RL ground. 2. **Separate late from unattributable.** A late but attributable outcome is a supervised problem with a lag. 3. **Stress-test the proxy.** Name the behaviour that would maximise the proxy while hurting the objective. If that behaviour is reachable, the proxy is unsafe. 4. **Only then ask about cost.** Delayed credit establishes that RL is the right *shape* of solution; it says nothing about whether you can afford the interaction to train one. Many problems are honestly RL-shaped and still get solved with a supervised model plus a hand-written policy, because the interaction budget does not exist. ## What a good answer sounds like A strong candidate does not argue that RL is better because it optimises the long run. They start from the label, show why it is missing, name credit assignment as the specific gap, and then volunteer the cheaper alternative and the condition under which it breaks. The willingness to say "here I would supervise a proxy and monitor for the gaming pattern" is what marks judgment rather than enthusiasm.