How does a value function assign credit when the reward arrives only at the end of a long sequence?

## The problem The **credit assignment problem** is the question of which of many earlier decisions deserves credit for an outcome that arrives much later. It is the feature that separates sequential decision problems from ordinary supervised prediction. In supervised learning each input has its own label; in a sequential problem you get one number at the end of a long chain and must work out which links mattered. A concrete version: a marketing sequence sends five emails over three weeks, and the reward — the customer converts, worth some amount — lands after the last one, if at all. Every earlier send produced no revenue. Were they useless? Obviously not: without them the conversion would not have happened. But which ones, and how much? ## What a value function does about it The answer is built into the definition. The return from time t is `G_t = R_{t+1} + g*R_{t+2} + g^2*R_{t+3} + ...` — everything that comes afterwards, discounted. The value `V(s) = E[G_t | S_t = s]` is therefore not "what this state pays" but "what this state is worth given everything it leads to". A state that pays zero immediately can have a large value; a state that pays a small reward now but leads to a dead end can have a small one. The Bellman recursion is the mechanism that carries the terminal reward back. `V(s) = E[r + g*V(s')]` says the value of a state is one step of reward plus the discounted value of where you land. Apply it near the end and the state before conversion inherits the conversion reward. Apply it a step earlier and that state inherits a discounted share of the first. Credit propagates backwards through the chain, in proportion to how much each state actually raises the expected downstream reward. The important subtlety is that this is **not** an even split and **not** a rule of thumb. A send that barely changes the customer's trajectory produces a next-state value close to the current one, so it earns almost nothing; a send that moves the customer from "lapsing" to "engaged" produces a large jump in value and is credited accordingly. The credit is a consequence of the dynamics, not of a heuristic weighting. ## Contrast with attribution heuristics Business practice usually reaches for a fixed attribution rule: last-touch gives all credit to the final email, first-touch to the first, linear splits it evenly across touches. Each is a fixed prior on credit that ignores what the touches actually changed. Value functions replace the rule with a measurement: score each state by expected future reward, and the difference in value across an action is what that action was worth. This is also why the value framing is useful even when nobody plans to deploy an agent — the numbers answer the attribution question directly. ## Where it goes wrong **State representation.** Credit is assigned to *states*, so if the state does not capture what actually distinguishes two customers — how many emails they have already opened, how long since the trial started — different histories collapse into one state and its value is an average over them. Credit then gets attached to the wrong thing. Enriching the state is the fix, at the price of more states to estimate. **Variance and horizon.** The further the reward is from the action, the more independent randomness lies between them, so the return observed after an early state is a noisy sample of its value. With one conversion per customer, a single episode carries very little information about the first send. Long delays mean many episodes, and they mean the estimate for early states settles last. **Discounting.** The discount weights how far back credit travels. It is part of the problem specification and changes the answer: strong discounting concentrates credit near the outcome and can make genuinely important early actions look worthless. **Confounding.** A value function estimated from logged data credits states that *correlate* with conversion, which is not the same as states the sends *caused*. If the sequence only reaches engaged customers in the first place, high value on early states may reflect who was selected, not what the email did. ## What interviewers listen for That you can say why a zero-reward state can have a high value; that you name the Bellman recursion as the propagation mechanism rather than inventing a splitting rule; that you contrast it with last-touch attribution; and that you volunteer the statistical cost — long horizons make early-state values the hardest numbers in the model to pin down.