How does SAC's maximum-entropy objective differ from maximising return alone?

## The objective Ordinary control maximises `E[ sum_t gamma^t r_t ]`. The maximum-entropy formulation maximises E[ sum_t gamma^t ( r_t + alpha * H(pi(.|s_t)) ) ] where `H(pi(.|s)) = - E_{a ~ pi} [ log pi(a|s) ]` is the policy's entropy at state `s`, and `alpha > 0` is the **temperature**. The agent is paid both for reward and for keeping its action distribution spread out. This is not a tweak of the actor's loss function; it changes what "optimal" means. Under this objective the optimal policy is stochastic by construction: it concentrates only where concentrating actually buys reward, and stays broad where the reward surface is flat. ## Soft values: the entropy is inside the critic The important structural point, and the one candidates most often miss, is that the entropy term appears in the **bootstrap target**: a' ~ pi(.|s') (a fresh sample, not a stored action) y = r + gamma * ( min(Q1_target, Q2_target)(s', a') - alpha * log pi(a'|s') ) critic loss = (Q(s, a) - y)^2 for each critic Because `- log pi(a'|s')` is a single-sample estimate of the entropy at `s'`, the critic is learning the *entropy-augmented* return, sometimes called the soft Q-function. If entropy lived only in the actor's loss, the actor and critic would be optimising different objectives and the bootstrapped values would understate the worth of remaining stochastic. SAC also carries twin critics with a minimum in the target, for the same anti-optimism reason as any modern continuous-control critic. ## The actor's update The actor maximises E_{a ~ pi(.|s)} [ Q(s, a) - alpha * log pi(a|s) ] over states drawn from stored experience. Because the action is sampled, the gradient is taken by reparameterising: draw a fixed-distribution noise variable, push it through the policy's deterministic map to get the action, and differentiate through both the critic and the log-density. The first term pulls toward high-value actions; the second is a pull toward keeping the density low, i.e. spread out. `alpha` is exactly the exchange rate between one unit of reward and one nat of entropy. ## Auto-tuning the temperature A fixed `alpha` is a poor idea, because the reward scale differs per task and per training phase, and a temperature that is right early is usually wrong late. SAC instead treats the objective as a constrained problem — maximise return subject to the policy's average entropy being at least some target `H_target` — and adjusts `alpha` by gradient on the constraint violation. The behaviour is easy to state: - measured entropy **below** target -> `alpha` rises -> more pressure to spread out; - measured entropy **above** target -> `alpha` falls -> more pressure to exploit. `alpha` is optimised in log-space so it stays positive. The common heuristic sets `H_target` to minus the dimensionality of the action vector — one negative nat per action coordinate. That number is worth understanding rather than memorising: for a single coordinate bounded to the interval from -1 to 1, a uniform policy has differential entropy `log 2 ≈ 0.69`, so a target of -1 per coordinate asks for a policy substantially more concentrated than uniform while still meaningfully random. (Differential entropy can be negative; this is not a bug.) On a dexterous in-hand manipulation task — rotating a cube between fingertips — this shows up concretely. Early on the policy needs to be broad enough to discover regrasps at all. Later, the contacts are delicate and a policy that keeps jittering drops the cube; the auto-tuned temperature falls on its own as the return signal grows, whereas a hand-fixed temperature has to be re-tuned by a human at exactly that transition. ## What the entropy term buys, and what it costs Buys: - **State-dependent exploration.** Randomness where the value surface is flat, precision where it is sharp — a fixed injected noise level cannot do this. - **Robustness to critic sharpness.** A policy that must keep spread cannot commit hard to one razor-thin spike in the critic. - **Far less hyperparameter tuning**, since the exploration knob is optimised rather than scheduled. Costs: - The learned optimum is **not** the reward-optimal policy; it is the optimum of a different objective, and with a large `alpha` the gap is visible as a permanently sloppier controller. - Deployment needs a decision about whether to keep sampling or act at the distribution's centre. - The entropy estimate depends on getting the action density right, which is exactly where a bounded-action transform can silently corrupt everything.