What does the delta parameter control in Huber loss for a regression fit?

## The shape Huber loss is a deliberate compromise between the two standard regression losses. Writing the residual as `r = actual - prediction`: ``` L(r) = 0.5 * r^2 if |r| <= delta L(r) = delta * (|r| - 0.5 * delta) if |r| > delta ``` The constants are not arbitrary. At `|r| = delta` the first branch equals `0.5 * delta^2` and so does the second, so the loss is continuous; and the slope of the first branch at that point is `delta`, which is exactly the slope of the second, so the loss is also continuously differentiable. Huber is smooth everywhere, including at zero, which absolute error is not. ## Delta in the gradient The fitting behaviour is easier to read from the derivative with respect to the prediction: ``` gradient magnitude = |r| if |r| <= delta gradient magnitude = delta if |r| > delta ``` Inside delta the loss behaves like squared error: pull grows with the miss. Outside delta the pull is **clipped** -- a residual of `10*delta` and a residual of `1000*delta` exert exactly the same force on the parameters. That clip is the whole mechanism. Nothing is deleted, nothing is reweighted by hand; a badly wrong row simply loses the ability to buy influence by being wronger. So the honest one-line reading of delta is: **the residual magnitude above which a row is treated as an outlier for fitting purposes**. ## The two limits - **delta very large** relative to typical residuals: almost every row lands in the quadratic branch, and the fit is effectively a squared-error fit (scaled by one half). All robustness is gone. - **delta very small**: almost every row lands in the linear branch, and the fit approaches an absolute-error fit, up to a scale factor. You get maximum robustness, and you also give up the smooth behaviour near zero that made Huber attractive, plus some statistical efficiency when the data is clean. Delta therefore interpolates continuously between the two losses, and choosing it is choosing where on that dial to sit. ## Why it is scale-dependent, and how to choose it Delta is compared directly against residuals, so it carries the target's units. A trip-duration model in seconds and the same model in minutes need deltas differing by a factor of 60. Anything that changes the target's scale -- a unit change, standardising the target -- invalidates a previously tuned delta. This is the most common practical mistake with Huber: copying a delta across problems as if it were a unitless knob like a learning rate. Practical ways to set it: 1. **Tie it to a robust estimate of residual spread.** Fit, take a spread estimate of the residuals that is itself not dragged by the tail (a median-based deviation rather than a standard deviation, since the standard deviation is inflated by exactly the rows you are trying to tame), and set delta to a small multiple of it. A classical default is about 1.345 times that robust scale, a constant chosen so the estimator retains roughly 95 percent efficiency when the errors genuinely are Gaussian -- you pay only a few percent for the insurance. 2. **Iterate.** Residual scale changes as the fit improves, so a delta chosen from an initial fit is worth re-estimating once. 3. **Tune it by cross-validation** against whatever number the business actually judges the model on. If the evaluation criterion is itself tolerant of large misses, cross-validation will happily push delta down; if the evaluation criterion punishes them, it will push delta up. That is the correct feedback loop. A useful sanity check after fitting: what fraction of training rows sit in the linear branch? If it is essentially zero, delta is doing nothing and you have a squared-error fit with extra machinery. If it is half the data, delta is too tight and you are throwing away information from perfectly ordinary rows. ## A worked setting A ride-hailing trip-duration regression is trained on millions of trips that mostly run 5 to 40 minutes. A small number of trips carry GPS glitches and are recorded at nine hours. Under squared error those few rows have residuals in the hundreds of minutes and dominate the gradient, tilting the fitted surface for every ordinary trip. Set delta to something like a handful of minutes -- a few times the typical residual spread -- and each glitch row's pull is capped at that value. It still counts, in the right direction, but it counts once. The fit for the 5-to-40-minute mass of the data is restored, and the model is still smooth to optimise. ## What delta is not It is not a probability, not a percentile, and not a threshold on the target value -- it is a threshold on the **residual**, so whether a given row falls into the linear branch depends on the current fit and changes during training. And it is not a robustness guarantee against unusual feature values: like absolute error, Huber caps the influence of rows with surprising targets, not rows sitting far out in feature space.