How do you choose between pre-norm and post-norm for a model you intend to scale deeper?

## What the decision actually is Post-norm puts the normalization after the residual addition, `x_next = Norm(x + F(x))`; pre-norm puts it on the branch input, `x_next = x + F(Norm(x))`. Choosing between them is not a tuning knob you can revisit at step 50,000 — it is a commitment made before the first serious run, because the two compute different functions and no trained weights transfer between them. ## The evidence is genuinely two-sided The honest summary of the published picture is a depth-versus-quality framing: - At **moderate depth** — a handful of blocks — both placements train, and post-norm has been reported to reach slightly better final quality. - At **large depth** — on the order of a hundred blocks — plain post-norm becomes very hard to train at all, while pre-norm trains routinely. A candidate who reports only half of this is not being careful. The reason the field defaulted to pre-norm is not that it wins everywhere; it is that trainability is a precondition and quality is an optimization on top of it. A configuration that diverges in the opening steps of your target-depth run does not have a quality number to compare. So the decision rule is: **pick by the depth you intend to reach, not the depth you are prototyping at.** If a six-block prototype is a stepping stone to a much deeper model, tuning the prototype's placement on its own quality is optimizing the wrong objective — you will pay for it by having to redo the tuning after the switch. ## The comparison trap The most common analysis mistake here is an A/B run at a single fixed hyperparameter configuration. Placement changes what the model wants: how large a learning rate it tolerates, how much care the opening steps need, and how much explicit regularization is appropriate. Whichever placement your configuration was tuned for will look better, and the margin you measure is mostly the tuning, not the architecture. If you want a comparison that means something, tune each arm separately — at minimum sweep the learning rate for each — and report the best of each. Otherwise report the result as what it is: a comparison at the incumbent's operating point. ## The incidental regularizing effect Placement carries a regularization side effect that is easy to miss. Post-norm re-standardizes the main path at every block, which continuously constrains how far the representation's scale can drift; pre-norm deliberately leaves that path alone. So the two arrive at the same nominal weight decay and dropout settings with different effective regularization, and a team switching from post-norm to pre-norm often finds it needs a little more explicit regularization than before to land in the same place. This is another reason a placement swap is a retune, not a substitution: the tuned recipe you trusted was co-adapted to the placement it was tuned with. ## Middle grounds exist The choice is not strictly binary. Published variants keep the post-norm structure but modify the addition so that very deep stacks remain trainable — DeepNorm, for example, scales up the identity branch before the post-norm addition and pairs that with a matched initialization scale, and is reported to train stacks far deeper than plain post-norm manages. There are also designs that normalize both the branch input and the branch output before adding. These are worth knowing about, but they carry their own tuning obligations, and for most teams the interesting question is whether the extra machinery buys anything over pre-norm plus a final normalization. ## How to actually make the call 1. **State the target depth**, including where you expect to be two scale steps from now. This is the input that dominates. 2. **If the target is deep**, take pre-norm, and design in the final normalization before the head from the start. 3. **If the model is genuinely shallow and will stay shallow**, post-norm is defensible, and the reported quality edge is real — but budget for careful early-step handling. 4. **Decide once.** Put the rationale in the model card or design note so it is not re-litigated by whoever inherits the run. 5. **If you must compare**, tune both arms and say so; never ship a placement conclusion drawn from a single shared configuration. ## What an interviewer is listening for That you refuse the false choice between "pre-norm is better" and "post-norm is better", that you name trainability as a gate rather than a metric, that you know the swap invalidates tuning and the checkpoint, and that you would write the decision down rather than leave it as folklore.