How do you pick a gradient-clipping threshold instead of inheriting a default of 1.0?
answer
- the number depends on your model, not theirs
- collect before you choose
- look at the histogram of observed norms
- upper tail, not the body
- roughly one step in ten rescaled
basics
~20 sMeasure before you choose. Record the global gradient norm for the first several hundred steps with the clip effectively off, look at the distribution, and set the threshold near its upper tail — around the 90th percentile — so ordinary steps pass and only outliers get rescaled.
solid answer
~60 sA threshold is only meaningful relative to the norms your model actually produces, and that scale depends on the architecture, the loss reduction, the batch size and the parameterization — so a 1.0 copied from another project means nothing here. Run a few hundred to a thousand steps with the clip set high enough that it never binds, log the global norm each step, and look at the histogram. Set the threshold in the upper tail of that distribution, roughly the 90th percentile, so about one step in ten is rescaled and the body of the distribution passes through untouched. Then keep an eye on the fraction of steps that clip: if it drifts toward most steps, either the threshold is too tight or something about the run changed, and re-calibrating beats leaving it. The two failure modes bracket you — set it above every observed norm and it is a no-op that only feels safe; set it far below the typical norm and every step becomes the same length.
go deeper
Know that the clipping threshold is a number someone chose, not a constant of nature, and that it is compared against the size of the whole gradient rather than against the loss.
Be able to describe the measurement: run with the clip off, record the global norm each step, and place the threshold in the upper tail of the observed distribution rather than copying a default.
Show you monitor the fraction of steps that actually clip and re-calibrate when it drifts, and that you can name both failure modes — a guard that never fires and a threshold that turns every step into the same length.
Own the policy question: whether a shared default belongs in a team's training config at all, and what evidence a team should be required to attach before writing a threshold into one.
## Why the default number is meaningless A clipping threshold is a bound on the Euclidean norm of the whole gradient. That norm has units and a scale, and the scale is set by things that vary from project to project: how many parameters there are, whether the loss is averaged or summed over the batch, how large the batch is, how the targets are scaled, and how the layers are parameterized. A threshold of 1.0 might sit in the extreme tail on one model and below the median on another. Inheriting the number from someone else's configuration is inheriting a decision that was never about your model. ## The measurement The procedure is short and it is the answer an interviewer is listening for. 1. Turn the clip off, or set it so high it can never bind, and start a normal run. 2. Each step, after the backward pass and before the update, record the global gradient norm — the same scalar the clip would compare against. 3. Let it run for a few hundred to a thousand steps, long enough to get past the very first transient and to see the shape of the distribution. 4. Plot the histogram. You are looking for the body — where most steps live — and the tail. 5. Put the threshold in the upper tail. Around the 90th percentile is a good starting point: roughly nine steps in ten pass through untouched, and the tenth is rescaled rather than allowed to dominate. The shape matters as much as the number. A tight, unimodal distribution with a thin tail means clipping is cheap insurance and almost never fires. A heavy right tail — most steps around some value, with occasional excursions one or two orders of magnitude higher — is exactly the situation clipping exists for, and the percentile choice cleanly separates the two regimes. ## What the two failure modes look like **Set too high.** If the threshold sits above every norm the model ever produces, clipping is a no-op. Nothing breaks, the configuration line looks responsible, and the run has no protection at all. This is the more common mistake because it is invisible: nobody notices a guard that never fires until the step it should have caught. **Set too low.** If the threshold sits far below the typical norm, nearly every step gets rescaled to length exactly `c`. The update length is then constant, and the only thing carrying information from step to step is the direction. Magnitude — the signal that says *how strongly* the loss wants to move — has been discarded across most of training. You can view the effective learning rate as `lr * c / norm`, which shrinks exactly when the gradient is largest, so the phase where the model has the most to learn is also the phase you have throttled hardest. Training does not blow up; it just crawls, and the cause is not obvious from the loss curve alone. ## Keeping it honest over the run The norm's typical scale is not fixed for the life of a run. It usually changes as the model fits, and it changes if you alter the batch size, the loss weighting or the data mixture. So track one derived quantity: the fraction of steps on which the clip actually binds. That number is the calibration signal. A few percent means the threshold is doing what you designed it to do. A number that climbs toward most steps means the threshold has become a straitjacket — go back to the histogram rather than leaving it. A number stuck at zero for the whole run means you have no guard. Be explicit about which quantity you log, too: the norm *before* clipping is the one that tells you about the distribution. The post-clip norm is capped by construction and its histogram will pile up at the threshold, which tells you how often the clip fired but nothing about how far out the tail actually goes. ## When the percentile rule is the wrong tool If the histogram has two clearly separated modes — a body and a distant cluster of enormous norms — the percentile is a blunt instrument, and the interesting question is what produces the second cluster. Setting the threshold between the modes is a reasonable stopgap, but the calibration is now covering for a cause you have not identified. The threshold is a number you choose from evidence, not a way to avoid collecting the evidence.
- What does training look like when the threshold sits far below the typical gradient norm?Nearly every step gets rescaled to length exactly the threshold, so every update is the same size and only its direction varies. Magnitude information is thrown away, and the effective learning rate becomes `lr * c / norm`, which is smallest precisely when the gradient is largest. The run does not diverge — it just makes slow, uniform progress, and nothing in the loss curve says why.
- Should the threshold be revisited once training is under way?Yes. The scale of the gradient norm shifts as the model fits and changes outright if you touch batch size, loss weighting or the data mixture. Track the fraction of steps on which the clip binds: a few percent means it is behaving as designed, a number climbing toward most steps means it has become a constraint on ordinary training and the histogram is worth revisiting.
- Which norm should you log — before or after clipping?Before. The pre-clip norm is the quantity whose distribution you are trying to characterize, and it shows how far the tail actually reaches. The post-clip norm is capped by construction, so its histogram piles up at the threshold and tells you only how often the clip fired, never how extreme the underlying step wanted to be.
saying these in an interview costs you the question
- Says 1.0 is the standard threshold and stops there
- Picks a threshold without ever measuring a gradient norm
- Assumes a threshold transfers across models and batch sizes
- Thinks a threshold that never fires is proof of stability
- Logs the post-clip norm and reads it as the distribution