Would you swap GELU for ReLU in a battery-powered on-device vision backbone?

## Why this is a real question and not a micro-optimisation On a large model whose time goes into big matrix multiplications, the elementwise activation is usually a rounding error and nobody argues about it. On a small, battery-powered vision backbone the picture changes. The compute in the convolutions has been deliberately shrunk - fewer channels, cheap factorised convolutions - but the feature maps in the early stages are still large, because the input resolution is set by the camera and the task. The activation function runs once per element of every feature map, so its work tracks activation *volume*, which the network designer did not shrink nearly as much as the parameter count. That is how a parameter-free function ends up on the profile. GELU needs an error function per element (or a tanh-based approximation of it, which still costs an exponential); SiLU needs an exponential; ReLU needs a comparison and a select. The gap per element is large in relative terms even when it is small in absolute terms. ## The decision procedure **1. Measure before deciding.** Profile per layer on the target hardware, in the deployed precision, at the real input resolution, with the device in its real thermal and power state. Workstation numbers do not transfer: the relative cost of transcendental functions differs sharply across hardware. Report both latency per frame and energy per frame - a change can be latency-neutral and still shorten battery life, or vice versa. **2. Bound the prize.** If activations are a few percent of frame time, the entire swap can win at most a few percent, and you should go look somewhere else. If they are a meaningfully larger slice, continue. **3. Prefer the cheapest change that keeps the shape.** Before dropping smooth gates entirely, consider a piecewise-linear approximation of the gate. Hard-swish, published with the MobileNet line of mobile backbones, replaces the sigmoid gate with a clipped linear ramp, keeping the general shape while removing the exponential. It is the standard first move when the smooth unit is the problem and the shape is worth keeping. **4. Localise.** Activation element counts fall sharply as spatial resolution drops through the network. Using a cheap activation in the early high-resolution stages and a smooth one deeper costs little and keeps most of whatever the smooth unit was contributing. **5. Retrain, always.** A trained network's weights are fit against the nonlinearity that was present during training. Substituting a differently-shaped activation into fixed weights changes every layer's output distribution and typically degrades accuracy badly. Retrain from scratch, or at minimum fine-tune, and compare on the deployment metric rather than on training loss. ## Be honest about the benefit side The usual argument for smooth gates is that a continuous derivative through the origin gives a better-behaved local model of the loss than ReLU's kink, permitting somewhat larger stable steps and less erratic curvature. It is a reasonable argument, but it is an argument, not a guarantee. In practice ReLU's single non-differentiable point is handled with a subgradient and does not obstruct training, and the measured accuracy difference between ReLU and a smooth gate on a given small backbone is frequently within run-to-run variation. A senior answer says this plainly: treat the accuracy claim as a hypothesis to test with a seeded, repeated ablation on your own model and data, not as settled fact. ## Second-order consequences worth raising - **Deployment numerics.** ReLU's output is non-negative, which fits an unsigned activation range naturally. Smooth gated units emit a small negative range, so a reduced-precision deployment path must carry a signed range for those tensors. It is not a blocker, but it is a thing to check rather than discover late. - **Kernel support.** A fused, well-optimised implementation of a common activation on the target runtime can be much faster than a generic elementwise pass. Whether the shape you want is fused on your target is part of the cost, and it can invert the ranking you predicted from arithmetic alone. - **Consistency across a model family.** If a team ships several models to the same device, using one activation everywhere means one tuned kernel path and comparable baselines. That consistency is often worth more than a per-model activation choice. ## What a strong answer sounds like "I would not swap on principle either way. I would profile per layer on the device, see what share the activations actually take, and if it is worth chasing, try hard-swish first and reserve the full ReLU fallback for the case where I need the frame time badly. Whatever I pick, I retrain and compare accuracy against measured latency and energy, and I expect the accuracy gap to be small."