Your resale-price regression head predicts -30 for cheap items — how do you fix it?

## Why the negative appears A regression head is the simplest head there is: a linear layer of width 1 (or width d for a multi-output regression) applied to the trunk's features, with **no** activation on top. Its output is `w . h + b`, an affine function of the hidden vector, which ranges over all of the real line. There is no mechanism by which the architecture could know that resale prices are positive. If the trunk's features for a very cheap item extrapolate slightly past the cheapest examples the model saw, the linear head happily returns -30. A negative prediction is therefore a *modelling* omission, not a numerical failure. ## Four ways to respond **1. Leave it unbounded and clip at serving.** `max(pred, 0)`. Zero training cost, and it is the honest choice when negatives are rare, small, and confined to a region you do not care about. But the model is still being trained to produce impossible values, the gradient never learns that the region is forbidden, and everything clipped to exactly 0 becomes indistinguishable — you lose the ability to rank the cheapest items against each other. **2. Predict in log space.** Train on `log(y)` (or `log(1 + y)` if zeros occur) and exponentiate at inference. Positivity is guaranteed because `exp` cannot return a non-positive number. The deeper effect is on the error geometry: equal errors in log space are equal *ratio* errors in the original space, so being off by 10 currency units matters much more for a 20-unit item than for a 2000-unit one. For prices spanning orders of magnitude that is usually the behaviour you want, and it is the standard choice. The costs are that the exponentiated prediction is no longer an unbiased estimate of the mean in the original units, and that a large log-space error becomes an enormous absolute error after exponentiating. **3. Apply a positive-valued output transform.** Put softplus, `log(1 + exp(z))`, on the head's scalar so the output is positive by construction while staying close to linear for large `z`. This keeps the objective in the original units — useful when the business metric is an absolute currency error — while removing the impossible region. It is a lighter-touch fix than the log transform, and it does not change the relative weighting of cheap and expensive items. **4. Bound both ends with a scaled sigmoid.** For a quantity with a genuine finite range — an occupancy fraction constrained to [0, 1], a probability-like ratio, a normalised angle — apply a sigmoid to the head's scalar, and scale/shift it if the range is `[lo, hi]` rather than `[0, 1]`: `pred = lo + (hi - lo) * sigmoid(z)`. Now the prediction is structurally incapable of leaving the range, and every downstream consumer can rely on that without validation. ## What a bounded head costs you The bound is a hard architectural claim. If the true range is wider than you assumed — occupancy that can exceed 1.0 because of double-counting, or a price ceiling you set from last year's catalogue — the model cannot express the truth at all, and the error shows up as a flat line at the boundary rather than as a visible outlier. A bounded head also compresses: to output values very near either endpoint, the pre-activation must be driven far from zero, so predictions crowd towards the interior and the extremes are approached slowly. Impose bounds only where the domain genuinely guarantees them, and prefer a one-sided transform when only one side is physically constrained. ## Choosing between them Ask two questions. *Is the bound physical or empirical?* A fraction that is a ratio of counts is physically in [0, 1] — bound it. A price whose observed maximum happens to be 5000 is not bounded above at all — do not bound it. *What error geometry does the product want?* If a 10% error is equally bad at every price point, work in log space. If a fixed currency error is what the business loses, keep the original units and use a one-sided positive transform. ## Related head hygiene Standardising targets (subtracting the mean, dividing by the standard deviation) is orthogonal to all of this: it stabilises the scale the head must learn, but a standardised target is still unbounded, so it does not prevent negatives after you invert the transform. Multi-output regression heads follow the same rules per output — a head predicting width and height needs positivity on both, and each output may want a different transform. And whatever transform you pick belongs *inside* the served model, not in a caller's post-processing, so every consumer inherits the guarantee.