How does a trained weight matrix's singular-value spectrum tell you whether to factorize it?

## What the spectrum is telling you Every real matrix `W` has a set of non-negative singular values, conventionally sorted from largest to smallest. They describe how much the linear map stretches along each of a set of orthogonal input directions, mapped to a set of orthogonal output directions. A large singular value means the map does a lot along that direction; a near-zero one means the map barely acts there at all. Plotted in order, they form the matrix's spectrum, and the *shape* of that plot is the single most informative artifact when deciding whether a layer will survive being factorized. If the spectrum falls off a cliff - a handful of large values then a long tail of small ones - the layer is doing something that a rank-r matrix can imitate closely. If it decays slowly and every value is of similar magnitude, the matrix is well conditioned in the classical sense (largest over smallest is near 1), its map genuinely uses all its directions, and imposing a low rank throws away a proportional chunk of it. A well-conditioned layer refuses to factorize, and no amount of fine-tuning changes the fact that the starting approximation was poor. ## Turning that into a rank The usual quantitative criterion is retained energy. Squared singular values sum to the squared Frobenius norm of the matrix, so the ratio of the top-r squared values to that total is the fraction of the matrix's "mass" preserved by the best rank-r approximation, which is obtained by truncating the decomposition. Pick a threshold - many teams start at 95% or 99% - and read off the smallest r that clears it. A useful scalar shortcut is the stable rank: Frobenius norm squared divided by spectral norm squared, i.e. the sum of all squared singular values divided by the largest one. It answers "how many directions' worth of energy is here?" in one number, without a plot, and it is bounded above by the true rank. A layer whose stable rank is a small fraction of its dimensions is a candidate; one whose stable rank is close to full is not. Then apply the economic filter. Factorization only saves anything while the rank stays below `m * n / (m + n)`. Both facts must hold at once: the layer must *tolerate* a rank that is also *below break-even*. Plenty of layers tolerate rank 700 out of 1024 - and rank 700 in a 1024-square layer costs more than the dense original. That layer is not factorizable in any useful sense even though 95% of its energy fits. ## Why the spectrum is a shortlist, not a verdict The energy criterion measures error in *weight space*: how far `A B` is from `W`. What you actually care about is error in *task space*: how far the network's outputs and loss move. These differ for two reasons. First, inputs are not isotropic. A direction with a small singular value can still matter enormously if the activations that hit this layer have large variance along it, and a direction with a large singular value can be nearly irrelevant if nothing ever excites it. This is why data-aware factorization exists: instead of minimising weight-space error, you minimise the change in the layer's *outputs* on a calibration batch, which weights directions by how often and how strongly they are actually used. That routinely finds a lower usable rank than the plain spectrum suggests. Second, errors compound. A layer whose output shifts slightly feeds a layer whose behaviour was tuned to the unshifted distribution, and small per-layer errors can accumulate or interact non-linearly by the time they reach the loss. A model with ten layers each at 99% retained energy is not a model at 99% of its accuracy. ## The workflow that actually works 1. Count parameters per layer and rank layers by size. There is no point analysing a layer holding 0.3% of the model. 2. Compute each big layer's spectrum, its retained-energy curve and its break-even rank. Discard layers whose required rank sits above break-even. 3. Run a per-layer sensitivity sweep: factorize exactly one layer at a time at a couple of candidate ranks, measure validation loss with everything else untouched. This produces an empirical accuracy-per-parameter curve per layer, which is the thing you should be allocating against. 4. Allocate the compression budget non-uniformly - cheap layers get aggressive ranks, sensitive ones get left alone or barely trimmed. Uniform ranks across a network are almost always the wrong answer. 5. Fine-tune the whole factorized model to recover, and re-measure. Recovery capacity differs per layer too, so a layer that looked bad before fine-tuning may be fine after it, and the ordering can change. ## Which layers usually win The widest, most over-parameterized matrices - large hidden-to-hidden projections, oversized classifier heads, big input tables - both hold the parameters and tend to have the most redundant spectra. Narrow layers lose twice over: they hold few parameters, and their break-even rank is so small that the usable window barely exists.