For a bounded physical plant, would you deploy a deterministic actor with injected exploration noise or a maximum-entropy stochastic policy?

## Frame the decision correctly This is not "which algorithm scores higher". Both families are off-policy continuous-control learners with a critic; both bound actions by squashing into the physical range; both can be made to work. The decision axes that actually matter on a plant are: **where exploration comes from**, **who tunes it**, **what is shipped**, and **what stands between the policy and the actuator**. ## Axis 1 — the source of exploration A deterministic actor emits one action per state, forever. Exploration is therefore an external component: noise added to the action during data collection, with a chosen magnitude, a chosen temporal correlation (independent jitter explores very differently from a slowly drifting offset — on a thermal plant only the latter produces a meaningful excursion), and a chosen decay schedule. An entropy-regularised policy has exploration inside the objective. The agent is paid to keep spread, so it keeps it where spread is cheap and gives it up where reward demands precision. The behaviour is state-dependent for free, and the temperature that prices it is auto-tuned against a target entropy rather than scheduled by hand. On a cooling controller this difference is concrete. Fixed-scale noise applies the same excursion to a valve whether the hall has ample thermal headroom or is minutes from an alarm. State-dependent spread is not a safety mechanism, but it is at least *correlated* with how sharply value varies, which fixed noise is not. ## Axis 2 — the tuning bill The deterministic family is the one that costs engineer-days: noise scale, decay, and a critic that is fragile to it. If you operate one plant with a patient team, that is affordable. If you operate fifty heterogeneous plants, per-site noise scheduling is the dominant lifetime cost, and an auto-tuned temperature is the argument that wins. Be honest, though, that auto-tuning replaces the noise schedule with a **target entropy** — one knob instead of three, and a scale-free one, but not zero knobs. A candidate who claims the temperature removes all tuning has oversold it. ## Axis 3 — what you actually ship A plant operator does not want a jittering setpoint in steady state. Whichever family trains the policy, the deployed controller should be a single action per state: - deterministic actor: ship it as-is, noise off; - stochastic policy: ship the **median** action — the squashed centre of the pre-squash distribution — rather than a sample. Because the squash is monotone it preserves quantiles, so that value is available in closed form and is the natural point estimate. If you truly want live exploration on a production plant, that is a separate and much heavier decision requiring a bounded excursion budget and an interlock, not a side effect of leaving sampling switched on. ## Axis 4 — bounds are not a safety case Both families bound their outputs by construction, and it is tempting to call that safe. It is not. The squash guarantees the action lies in the interval you declared; it guarantees nothing about the *sequence* of actions, the rate of change, or behaviour on states outside the training distribution. On a real plant you want, outside the learned policy: - a per-step rate limit on each setpoint; - a supervisory envelope that overrides the policy on measured plant state; - a fallback controller with a defined handover, and monitoring that triggers it. Adding a movement penalty to the reward is worth doing as well, but it is a preference, not a guarantee — the guarantee has to live outside the network. ## Where the deterministic option genuinely wins - **Auditability and reproducibility.** A fixed function from state to action is easier to certify, diff between releases, and reason about than a policy defined as the centre of a learned distribution. If a regulator or a safety review is in the loop, that is real. - **Exploration confined to simulation.** If all data collection happens in a simulator, the noise schedule is a training-time detail nobody has to defend, and the argument about state-dependent exploration on the plant evaporates. - **An existing, tuned pipeline.** A team that already has a working noise schedule for this class of plant should not throw it away for an argument about elegance. ## The answer to give Default to the entropy-regularised learner for training because it removes per-plant exploration tuning and explores in a state-dependent way; deploy it deterministically at its median action; put the rate limiter, envelope and fallback outside the policy where they belong. Switch to the deterministic actor when the shipped artefact must be a fixed auditable control law, or when all exploration is confined to simulation and a tuned schedule already exists. State the trade explicitly rather than declaring one algorithm better — that is what the question is testing.