Model Efficiency and Compression

Factorize-then-fine-tune when trained weights exist and compute is short: the spectrum picks a per-layer rank and a brief retrain recovers most accuracy. Train the factorized shape from scratch when you want low rank learned, not imposed.

An architecture search pays when its cost amortizes: unusual target hardware, or one space reused across many deployment targets. For a single model on ordinary hardware, scaling a well-tuned published design is cheaper and often as good.

Spend bits where the sensitivity curve is steep and take them from where it is flat, equalizing the accuracy lost per bit saved until the size budget is met. Then validate the joint configuration, because per-layer curves were measured independently.

Treat the lottery-ticket result as a claim about trainability, not a deployment technique. Finding the mask means training the dense network repeatedly, and the artifact is scattered sparsity that ships no faster than dense.

The answer turns on what you still have: pruning plus fine-tuning is the cheap path, and the only one available without the original data and recipe. For a large cut, a purpose-built narrow architecture trained to convergence usually wins.