How do you assign per-layer bit-widths across a network under one total size budget?

## Framing You are given a total size budget — the compressed model must fit some fixed number of megabytes — and a network of, say, twenty layers. A uniform bit-width satisfies the budget but ignores everything you know: the sensitivity sweep already told you that some layers collapse at 4 bit while others are indistinguishable from full precision. The job is to convert that profile into a per-layer bit-width vector that meets the same budget at a smaller accuracy cost. ## The allocation rule Write each layer's size as parameters times bit-width, and let its sensitivity curve give an estimated accuracy drop for each candidate width. Two equivalent ways to state the optimum: **Greedy / equal marginal cost.** Start everything at the highest width you would consider. Repeatedly remove one step of precision from whichever layer loses the least accuracy per bit of size saved. Stop when the budget is met. At the stopping point the marginal accuracy-per-bit is roughly equal across layers — if it were not, you could move a bit from a cheap layer to an expensive one and come out ahead. **Lagrangian sweep.** For a multiplier lambda, choose for each layer independently the width minimizing `drop_l + lambda * size_l`. Small lambda gives a large accurate model, large lambda a small degraded one. Sweep lambda until the total size lands on the budget. This is the same optimum, computed in one pass per lambda instead of step by step, and it makes the tradeoff curve — accuracy against total size — fall out for free. The practical effect is exactly what the leaf name suggests: the flat layers donate the bits that the steep layers spend. A twenty-layer network might end with a handful of layers at 8 bit, the bulk at 4, and a couple of very redundant wide layers lower still, at the same total size as a uniform assignment that was measurably worse. ## The constraints that make it tractable - **The width set is small and discrete.** There is no point optimizing over a continuum: only a few widths are actually representable and worth supporting, so each layer has perhaps three or four options and the search is tiny. - **Granularity may be coarser than one layer.** If a whole block or a repeated stage must share a setting, optimize over groups, not layers. Fewer knobs also means a vector a human can read. - **Boundary layers are usually pre-assigned.** The stem and the output head typically start at a higher width, which removes them from the search and costs little budget. ## Where the arithmetic misleads you The curves were measured one layer at a time. Two consequences: **Drops do not add.** When ten layers are degraded simultaneously, the perturbations propagate and interact, and the joint drop is typically worse than the sum of the individual drops. The sum-of-independent-drops objective is a ranking heuristic, not a prediction of the final accuracy. Any chosen vector must be evaluated as a complete configuration before it is believed. **The profile shifts after retraining.** Recovery fine-tuning changes which layers are fragile, because layers adapt to the perturbation their neighbours now carry. That argues for one or two outer iterations — allocate, cut, retrain, re-measure the layers you cut hardest, reallocate — rather than a single-shot optimization polished to four decimal places on a model that no longer exists after training. ## The judgment call an interviewer is really probing A per-layer bit vector is a liability as well as an asset. It has to be re-derived when the architecture changes, when the data drifts, when someone adds a block. Against it stands the two-tier default: boundary layers at 8 bit, everything else at 4, decided in an afternoon and understandable by anyone. So the leadership answer is conditional. Do the full search when the model is a long-lived asset shipped at scale, when the uniform assignment misses the accuracy bar and the two-tier rule does not close the gap, or when the sensitivity profile is strongly uneven — a nearly flat profile means there is little to win and uniform is correct. Skip it when the architecture is still moving, when the team cannot commit to re-running the sweep on every change, or when the accuracy headroom is comfortable. If you do adopt a per-layer vector, ship it as versioned configuration alongside the profile it came from and the evaluation that validated it, name an owner for regenerating it, and record the accuracy at the uniform baseline so anyone can see what the complexity bought. An unowned, unexplained bit vector inherited across three model generations is worse than the simple rule it replaced.