Learning-Rate Range Test

## Why the test exists The learning rate is the single hyperparameter that decides whether a run learns at all. Too small and the loss barely moves for hours; too large and it climbs, oscillates or blows up. The usable band spans several orders of magnitude, and it moves with almost every change to the recipe, so guessing costs whole runs. A range test buys a measurement of that band for the price of a few hundred steps. ## The procedure Start from the initial weights of the model you actually intend to train, and save them first. Set the rate to something certainly too small — `1e-7` is a common floor — and pick a ceiling certainly too large, say `10`. Choose a step count N, then after every step multiply the rate by the constant factor `(max/min)^(1/N)`, so the rate traverses the range geometrically rather than in equal increments. Take one mini-batch per step, perform one update, record that batch's loss, and continue until you reach the ceiling or the smoothed loss exceeds its best value by a large factor — a stop rule of roughly four times the best loss is common, because a diverged model produces nothing readable afterwards. On the keyword-spotting model over 16 kHz voice-command clips, a sweep from `1e-7` to `10` across 200 steps is a typical shape: 200 steps across eight decades of rate is about 25 measurements per decade, enough to see structure. When the sweep ends, throw the weights away and reload the saved initial state. The sweep has taken a couple of hundred updates at rates ranging from useless to explosive, several of them past the point of blow-up; the model it leaves behind is damaged, not warmed up. The only artifact you keep is the number. ## Reading the plot Plot loss on the y-axis against rate on a logarithmic x-axis, and smooth the loss first — an exponential moving average over the per-step values — because a single mini-batch's loss is noisy enough to invent dips that are not there. Four regions appear: 1. **A flat left region.** The rate is so small that nothing measurable changes. 2. **A descent.** The loss falls, and somewhere in here it falls fastest. 3. **A minimum.** The smoothed loss stops improving. 4. **A sharp rise.** Updates overshoot and the run diverges. The trap is the minimum. It is the most visible feature, so it is the one people point at, and it is already too hot. Two reasons. First, the curve is a *trailing* measurement: the loss recorded at rate r reflects every earlier, smaller update too, so part of the improvement visible there was earned at lower rates. It is not the loss you would obtain by training at r. Second, by the time improvement stops, the rate is essentially at the edge of the region where updates still help; a full run held there has thousands more steps in which to meet a batch that overshoots. The two standard readings are the point of steepest descent — where the smoothed curve falls fastest — and roughly one order of magnitude below the minimum. On a curve that bottoms near `3e-2`, both rules land somewhere around `1e-3` to `3e-3`. ## How long the sweep has to be Count steps per decade, not steps. A 40-step sweep across eight decades is five points per decade, each point a single noisy mini-batch drawn from a million-example corpus: the result is jagged noise in which any dip can be misread as a knee. A few hundred steps over the full range is the usual working point. It is also normal to run a coarse sweep, see roughly where the interesting region is, then re-run over three decades with the same number of steps for a much cleaner curve. ## What the number is and is not Treat the output as an upper bound plus a sensible starting point, not an oracle. It is a short-horizon statement about training loss on one exact configuration. Confirm it on the first hundred steps of the real run: the loss should fall steadily rather than flatline or spike. And remember that the curve belongs to the whole recipe — the same architecture with a different batch size or a pretrained encoder produces a different curve and a different answer.