When would you share one encoder trunk between the actor and critic instead of training two networks?

## The two designs In the **shared** design, one encoder maps the observation to a hidden representation, and two small heads sit on top: a policy head producing a distribution over actions, and a value head producing a single scalar. In the **separate** design, two independent networks each take the raw observation and produce their own output, and nothing is tied. ## The argument for sharing Consider a datacenter job scheduler whose observation is the current queue - hundreds of pending jobs with resource requests, ages and priorities, plus per-machine utilisation. Encoding that into a useful representation is nearly all the work; deciding where to place the next job, given a good representation, is a small head. Two arguments follow: 1. **Cost.** The encoder dominates parameters, memory and forward-pass time. When you are batching observations from many parallel simulators, one encoder rather than two is a real, immediate saving in both memory and throughput. 2. **Signal quality.** The value head's target is a regression on returns, which is dense and comparatively low-variance. The policy gradient early in training is a high-variance signal built on near-random advantages. The value head therefore teaches the encoder useful structure - which queue states are dangerous, which are comfortable - before the policy gradient has anything coherent to say. ## The argument against sharing Both heads' gradients sum into the same encoder, and there is no guarantee they want the same features. In the joint loss - a policy term, plus a value coefficient times the value loss, minus an entropy coefficient times entropy - that value coefficient is not a cosmetic knob. It sets how much of the encoder serves value regression rather than control. The failure mode to have lived through: value loss grows with the square of the error in predicted return, and return magnitudes grow as the agent improves. On the scheduler, early returns are small and the two terms are comparable; a few hours in, the agent is completing far more jobs, returns are an order of magnitude larger, the squared value error is two orders larger, and the same fixed coefficient now means the encoder is essentially a return predictor with a policy head attached for decoration. The visible symptom is subtle and easy to misread: total loss keeps falling, value loss dominates it, policy entropy goes flat, and the return curve stops climbing. People reach for the learning rate, which is the wrong lever. ## What to do about it In rough order of what to try: 1. **Normalise the return scale.** Keep a running mean and standard deviation of returns and train the value head on the normalised target. This removes the drift at its source and is more robust than chasing the coefficient. 2. **Compare gradient contributions, not loss values.** The number that matters is the norm of each head's gradient measured at the top of the trunk. Two loss values on different scales tell you nothing about who is steering. 3. **Clip the global gradient norm.** This bounds the damage from a value spike but does not fix a persistent imbalance. 4. **Stop the value gradient at the trunk.** The value head then trains on features it does not shape. You keep the compute saving and lose the feature-shaping benefit - a reasonable trade once the encoder is good. 5. **Split the networks.** With a low-dimensional state the encoder is a couple of small layers, sharing saves almost nothing, and separate networks also let you run a larger learning rate on the critic, which usually wants to move faster than the actor. ## How to decide The honest decision rule is about where the cost and the risk actually are. - High-dimensional observations, expensive encoder, many parallel actors: share, and instrument the balance from day one. - Small state vectors, cheap networks: separate, and spend your tuning budget elsewhere. Sharing a trunk to "save parameters" on a ten-dimensional state is a coupling you are buying for nothing. - Anything in between: start separate, because it has fewer failure modes and gives you a clean baseline, then share if profiling says the encoder is the bottleneck. The part worth owning as a lead is that this is not a one-time architecture choice. A shared trunk has a hyperparameter that silently changes meaning as the agent gets better, and unless someone is watching the relative gradient contributions, the run degrades in a way that looks like a policy problem and is actually a plumbing problem.