What do process reward models reward that outcome reward models don't?

## The problem both are solving When a model produces a twenty-line derivation and you know only whether the last line is right, you face a credit-assignment problem: which of those twenty lines deserved the reward? Reinforcement learning from a single terminal signal has to spread that one bit across the whole trajectory, which is slow and noisy. Outcome and process supervision are the two answers to this, and the difference between them is where the reward signal is attached. ## Outcome supervision An outcome reward model, or a programmatic checker, produces one scalar per solution derived from the final answer alone. Its great virtue is that the label is often free and objective: for arithmetic, symbolic maths, unit tests, or anything with a canonical answer, you do not need a human or a learned model at all — you check the answer. This is what makes it scale to millions of samples, and it is the backbone of reinforcement learning with verifiable rewards, the dominant recipe behind the current generation of reasoning models. What it incentivises is answer-getting. Any chain that reliably produces the right final answer is reinforced, whether it is a clean derivation, a memorised shortcut, a guess dressed up as reasoning, or a chain with two errors that cancel. The trace is unconstrained, so nothing pushes it toward being a truthful account of the computation. ## Process supervision A process reward model scores each step of the derivation, typically as correct, neutral, or incorrect, and a solution-level score is formed by aggregating those step scores. In a stoichiometry problem, the reward attaches to "convert 12.4 g of magnesium to moles", "apply the 2:1 mole ratio", and "multiply by the molar mass of the product" individually. A step that misapplies the mole ratio is penalised at the point it happens, even if the arithmetic downstream rescues the final number. The signal is denser, so credit assignment is far better: the model learns which move was the mistake, not merely that the episode failed. It also directly rewards derivations a human grader would accept, which is what you want when the reasoning itself is the deliverable. ## What each one silently encourages This is the part interviewers care about. Outcome supervision tolerates — and, at scale, actively selects for — reasoning that is not a faithful account of the computation, because nothing in the objective examines it. Process supervision constrains the trace, but it introduces its own hazard: the model is now optimising against a learned judge of steps, so it can learn to write steps that the reward model likes. Confident, conventional-looking, well-formatted derivations score well whether or not they are sound, and the failure is harder to notice than a wrong answer because everything reads plausibly. There is also a subtler asymmetry. A process reward model must decide what an "incorrect" step means. Is a step wrong because it is mathematically false, or because it leads somewhere unproductive? Those are different labels, and the second one penalises legitimate exploration and backtracking — which is exactly the behaviour long-form reasoning models were trained to do. ## Cost and label sourcing Outcome labels: free wherever answers are checkable, otherwise a human or judge model on the final answer only. Process labels: expensive. A published human-labelled step dataset for competition maths runs to roughly 800,000 step labels, which gives a sense of the annotation bill. The cheaper route samples several continuations from each prefix and labels a step by how often it still reaches the right answer, requiring no human at all but inheriting the ambiguity above. ## Where the field actually stands As of mid-2026 this is not settled. Verifiable outcome rewards carried most of the recent gains in reasoning models, largely because they scale without annotation. Process rewards remain valuable as verifiers and as evaluation instruments, and in domains where no automatic answer check exists, but claims that one strictly dominates the other should be treated with suspicion. The honest framing in an interview is: outcome supervision is cheap, scalable and blind to the trace; process supervision is expensive, dense, and only as good as the step labels you can afford.