Rescaling a ReLU network leaves its function identical but inflates measured sharpness — what does that break?

## The construction The rectified linear unit satisfies ReLU(c*z) = c*ReLU(z) for any c > 0: it is positively homogeneous of degree one. Take two consecutive layers in a ReLU network, with the first computing h = ReLU(W1 x + b1) and the second computing W2 h. Now form a rescaled parameter set: W1' = c * W1, b1' = c * b1, W2' = W2 / c The hidden activation becomes ReLU(c*W1 x + c*b1) = c * ReLU(W1 x + b1) = c*h, and the second layer multiplies it by W2/c, returning exactly W2 h. Every output for every input is bit-for-bit the same function. Training loss, validation loss, test accuracy, calibration — all unchanged, because the map from inputs to outputs is unchanged. ## Why the sharpness number moves anyway Sharpness is measured in parameter space, and the two parameter vectors are far apart even though the functions coincide. Consider a fixed-radius probe that adds a perturbation of absolute size delta to every weight. In the rescaled network the second layer's weights are c times smaller, so a perturbation of size delta is a *relatively* enormous change to them — and those weights now multiply an activation that is c times larger. The loss therefore climbs much faster per unit of absolute perturbation. Formally, the Hessian is transformed by the reparameterization and its eigenvalues along the affected directions grow; by choosing c large you can make the same solution look arbitrarily sharp. This is not a subtlety to file away. It means that a fixed-radius perturbation measure, the Hessian spectral norm and any other raw curvature statistic are properties of the *coordinates*, not of the function. Whatever they are correlated with in a typical experiment, they cannot be the mechanism, because they can be moved without touching the thing they are supposed to predict. ## What this does and does not refute It does not say flat solutions are unrelated to generalization. Within a fixed parameterization and a fixed training recipe, weight scales are not arbitrary — they are set by initialization, weight decay and the optimizer — so the comparison between two checkpoints from the same pipeline is usually meaningful, and the empirical correlation people observe is real. What it refutes is the strong claim: that sharpness is *the* explanation, and that a curvature number can be read off any two arbitrary models and used to rank them. Two networks from different architectures, different normalization choices or different weight-decay settings can differ in raw sharpness for reasons that have nothing to do with the functions they compute. The same degeneracy appears wherever the loss surface has scale symmetries. When a layer's output passes through a normalization that removes its scale, multiplying that layer's weights by a constant leaves the function unchanged too — so the parameterization is again not identified, and raw curvature in those directions is not meaningful on its own. ## The repair Make the measure invariant to the symmetry. Instead of perturbing every weight by the same absolute amount, perturb each coordinate proportionally to its own current magnitude — an element-wise adaptive radius. Under the rescaling above, the perturbation applied to the shrunken layer shrinks with it, and the measured sharpness stays put. The same idea can be applied to a sharpness-penalizing training objective, whose radius otherwise means different things in different layers and transfers badly between models. Other partial repairs: normalize by the weight norm and report a relative rather than absolute sharpness; restrict comparisons to checkpoints with matched weight norms; or report the whole radius sweep rather than a single number, so a reader can see whether two curves differ in shape or only in scale. ## The judgment call The practical stance a lead should hold is this. Sharpness is a useful *diagnostic within a controlled comparison* — same architecture, same recipe, same weight scale — and a bad *model-selection score across arbitrary models*. If someone proposes gating a release on a curvature number, ask what makes that number invariant, ask whether the candidates have comparable weight norms, and insist on a held-out metric as the actual decision criterion. And if someone dismisses flatness entirely because of the rescaling counterexample, the correct reply is that the counterexample indicts the *measure*, not the intuition: fix the measure, keep the intuition, and never let either replace an evaluation on held-out data.