When should sparsity be ramped up gradually during training rather than cut once after training?

## Two schedules for the same target Both procedures end in the same place — a masked network at a chosen sparsity — but they get there differently. **One-shot.** Train to convergence. Compute the threshold for the full target sparsity. Zero everything below it in one step. Fine-tune under the fixed mask until accuracy recovers or stops improving. One pruning event, one recovery phase. **Gradual.** Define a sparsity *schedule*: a function from training step to target sparsity, rising from zero (or from some initial level) to the final target over a span of training, then held flat for a final stretch. At intervals, recompute the mask so that the current target is met — this always means zeroing more weights, since the target only rises — and otherwise train normally. The commonly used ramp shape prunes aggressively at the start, when the network has the most redundancy and the most remaining training in which to recover, and slows as it approaches the final sparsity, when each additional weight removed is more costly. **Iterative prune-and-retrain** sits between them: prune a slice, retrain to convergence, prune again, repeat. It is the most expensive of the three and generally the strongest at extreme sparsity, because every cut gets a full recovery. ## Why gradual wins where it wins Recovery is a training problem. After a cut, the surviving weights must move to compensate, and how far they can move depends on how much training remains and how large the learning rate still is. A single cut to 95 percent removes so much of the function at once that the fine-tune starts from a badly degraded point — often one from which the optimiser cannot climb back, because the loss surface it now lives on is not a small perturbation of the one it was trained on. A sequence of small cuts, each followed by real training, keeps the model near a good solution the whole way. The empirical signature is a sparsity sweep. Take a text-classification encoder and evaluate at 50, 80, 90, 95 and 99 percent global sparsity under both schedules. At 50 percent the two curves sit on top of each other — the model has that much slack and one-shot plus a short fine-tune reclaims it. At 80 percent a gap opens. At 90 the gap is clear. At 95 and 99 one-shot has typically fallen off the cliff while gradual is still degrading smoothly, until it too finds its knee at a higher sparsity. **The value of the schedule is precisely that it moves the knee.** ## The learning-rate trap The most common way a one-shot cut is judged unfairly: pruning at the very end of a decayed schedule and fine-tuning at the final, tiny learning rate. At that rate the surviving weights barely move, so almost nothing is recovered and the conclusion drawn is that the model cannot tolerate the sparsity. Recovery needs a learning rate large enough to reorganise — restarting at a meaningfully higher rate with its own decay, rather than continuing the exhausted tail of the original schedule. Before concluding that a sparsity level is unreachable, check that the recovery phase had room to work. ## Choosing in practice Drive the decision from three inputs. 1. **Target sparsity relative to the knee.** Below the knee, take the cheap option. Near or beyond it, gradual is what buys the level. 2. **Compute budget.** Gradual pruning costs roughly a training run; if you are pruning a model you did not train and cannot afford to retrain, one-shot plus fine-tuning is the realistic plan and you accept a lower reachable sparsity. 3. **How many times you will do this.** Gradual pruning bakes the schedule into training, so re-targeting a different sparsity means another run. One-shot lets you produce several sparsity points from one trained checkpoint cheaply, which is what you want while you are still exploring where the knee is. A reasonable default: sweep sparsity with cheap one-shot cuts to locate the knee, then commit to a gradual run only for the level you intend to ship. ## Details worth stating - The last stretch of a gradual schedule should hold the final sparsity fixed while training continues, so the model ends on pure recovery rather than on a fresh cut. - The mask is monotone: because targets only rise and pruned weights are held at zero, a weight removed early does not return. - Whichever schedule you use, the mask must be reapplied after every update, or the optimizer will refill the zeros.