Why does TD3 train two critics and take the smaller of their target values?

## Where the overestimation comes from A critic is a regression model, so its output is the true value plus error, and that error is not tiny in the regions the agent has barely visited. Now add an actor whose entire job is `argmax_a Q(s, a)`. Maximising a noisy function does not return the maximum of the true function — it returns whichever point had the largest positive error. Formally, the expectation of a maximum is at least the maximum of the expectations, so the maximisation step is *systematically* biased upward even when the critic's errors are zero-mean. With a deterministic actor the maximisation is not a one-off: it is a trained network that spends every update searching for the critic's optimistic patches, and it does so faster than the critic can be corrected by new data. ## Why it compounds The critic bootstraps. The target for a transition `(s, a, r, s')` is `r + gamma * Q(s', a')`, where `a'` is the actor's action at the next state — precisely the action the actor has been optimising to maximise `Q`. So the inflated value at `s'` becomes the regression target at `s`, which inflates `s`, which inflates its predecessors. On a locomotion benchmark this shows up as an unmistakable diagnostic: the mean predicted Q on batches from the buffer rises smoothly and without bound, while the episode return plateaus and then falls. The critic has drifted into fantasy values, and the actor is faithfully chasing them. ## The fix: clipped double Q-learning TD3 instantiates **two critics**, `Q1` and `Q2`, with independent random initialisations. They see the same data and are trained on the same shared target, and that target uses the smaller of the two slowly-updated target critics: a' = actor_target(s') (plus smoothing noise, see below) y = r + gamma * min( Q1_target(s', a'), Q2_target(s', a') ) loss = (Q1(s, a) - y)^2 + (Q2(s, a) - y)^2 The two critics stay different only because of their initialisation and their independent optimisation paths — that is enough. The intuition is a vote for pessimism: for the minimum to be optimistic, *both* networks must be optimistic about the same action at the same state, which is far rarer than either one being optimistic alone. Note that the actor is trained against one designated critic (conventionally `Q1`), not the minimum; the pessimism is what matters in the target that drives learning. ## Why deliberate underestimation is acceptable The minimum is not unbiased — it pushes the estimate low. That is accepted on purpose because the two biases behave very differently under a maximising actor: - **Overestimation is self-amplifying.** The actor seeks out the overvalued action, so the wrong value gets rehearsed in every target and the error grows. - **Underestimation is self-correcting-ish.** An undervalued action is one the actor avoids, so the error simply stops being reinforced; the worst case is that a good action is explored less than it deserves. Asymmetric consequences justify an asymmetric estimator. The failure mode of too much pessimism is real, though — pushing the idea to many critics and taking the minimum over all of them makes the agent timid and slow to commit, which is why two is the standard choice rather than five. ## The two companions in the same algorithm The twin critics are one of three changes, and an interviewer usually wants all three: - **Delayed policy updates.** The actor and the slow target copies are updated once per two critic updates. The actor is then optimising against a critic that has had time to settle, rather than chasing a value surface that is itself moving fast — which reduces the chance of locking onto transient errors. - **Target policy smoothing.** Small clipped noise is added to the action used in the target, so the target value is effectively an average of `Q` over a neighbourhood of that action rather than its value at one exact point. Without it, a narrow spurious spike in the critic is a perfectly good thing for the actor to climb; a quadruped will discover a razor-thin peak in the critic and produce a gait that scores wonderfully and moves nowhere. Smoothing enforces the reasonable prior that similar actions should have similar value. ## Diagnosing it in practice Log the mean predicted value on replayed batches alongside the mean observed discounted return. Healthy training keeps them in the same neighbourhood. A widening gap with the prediction on top is the overestimation signature, and it is the fastest single diagnostic in continuous-control debugging.