Scaling and Architecture Search

## The problem compound scaling solves You have a working convolutional network and a bigger compute budget. There are three obvious ways to spend it: - **depth** — more layers, - **width** — more channels per layer, - **resolution** — a larger input image. The usual practice was to pick one and push it. Compound scaling is the claim that this is the wrong shape of decision, and that the three dimensions should be scaled together in a fixed proportion. ## How the three dimensions cost compute A convolution layer's multiply-accumulate count is proportional to `channels_in * channels_out * kernel_area * output_positions`. Read the three dimensions off that expression: - Adding layers multiplies total cost **linearly** — twice the depth, twice the FLOPs. - Scaling width by a factor `b` multiplies *both* `channels_in` and `channels_out`, so cost grows as **`b^2`**. - Scaling input resolution by `g` multiplies the output height and width, so `output_positions` and cost grow as **`g^2`**. So the total compute multiplier of a scaled model is roughly `d * w^2 * r^2` relative to the base. Two consequences fall straight out. First, doubling width is four times the compute, not twice — a very common misstatement. Second, resolution scaling costs the same quadratic factor as width but adds **no parameters at all** to the convolution kernels; what it inflates is activations, hence peak memory during training and inference. ## The compound rule Compound scaling defines the scaled model by a single coefficient `phi`: ``` depth = alpha^phi width = beta^phi resolution = gamma^phi subject to alpha * beta^2 * gamma^2 ≈ 2, with alpha, beta, gamma >= 1 ``` Because of the quadratic terms in the constraint, the total FLOP multiplier is approximately `2^phi`. Setting `phi = 1` roughly doubles compute, `phi = 3` roughly multiplies it by eight, and so on. That is the whole point of the constraint: it turns "give me twice the budget" into a well-defined, reproducible change to all three dimensions at once. The procedure has two stages: 1. **Fix the ratios once.** With `phi = 1`, grid-search `alpha`, `beta`, `gamma` on the *small base* network under the constraint. This is cheap because the base model is small. 2. **Move `phi` only.** Every larger family member reuses the same ratios. You do not re-search the coefficients per size, and that is what makes the family a family rather than a set of unrelated models. ## Why joint scaling beats one dimension The dimensions are coupled, not independent budgets. - Raise **resolution** alone and the extra detail is available at the input but nothing downstream can use it: the receptive field of the unchanged stack now covers a smaller *fraction* of the image, and the channel count is unchanged, so the finer textures have nowhere to be represented. Accuracy climbs for a while, then flattens — while activation memory keeps climbing quadratically. On something like a satellite land-cover classifier, where the informative signal really is fine-grained, this is a seductive trap: doubling the tile resolution feels like it should always help, and the memory bill arrives long after the accuracy curve went flat. - Raise **depth** alone and you eventually hit optimization and diminishing-feature-reuse limits; the marginal layer sees the same coarse input. - Raise **width** alone on a shallow stack and you get many channels of relatively low-level features. Each dimension saturates on its own, and each saturation is relieved by moving the other two. Scaling them in proportion keeps the network balanced, so the accuracy-per-FLOP curve of the family stays steeper for longer than any single-dimension curve. ## What it does not do Compound scaling is a **scaling rule, not an architecture**. It says how to grow a base network; the base network's quality still dominates the result. Apply the same rule to a poor baseline and you get an expensive poor model. The ratios are also tied to the base design and the data: they were found by grid search on one base network, and there is no guarantee they are optimal for a different block design or a much larger `phi` than the family was validated over. Two practical cautions. Activation memory, not parameter count, is usually the constraint you hit first when resolution grows, so "it only added 5% parameters" is not the reassurance it sounds like. And the scaling rule optimizes FLOPs by construction — if your real constraint is measured wall-clock latency on a specific device, a FLOP-balanced family is a starting point, not an answer.