Which skew transform handles a right-skewed column that contains zeros and negatives?

## The problem Many real numeric columns are right-skewed: most rows are small, a thin tail runs a long way out. Squared-error losses, distance-based methods and penalised linear models all react badly to that shape, because a handful of huge values dominate the fit. A monotone squeeze of the large values is the standard remedy. The trap is that the popular squeezes are not defined everywhere, and the column you actually have often contains zeros or negatives. ## The transforms and where each is defined **Natural log, `log(x)`** — defined only for `x > 0`. It compresses the tail hard: multiplying a value by 10 adds a constant to the transformed value, so the transform turns multiplicative spread into additive spread. Undefined at 0 and for negatives. **`log1p(x) = log(1 + x)`** — defined for `x > -1`, so it covers the whole of a non-negative column. It maps 0 to 0 exactly, which is why a zero-inflated column such as `lifetime spend` (a large point mass at zero, then a long right tail among spenders) is its classic use. For large x, `log(1 + x)` is almost `log(x)`, so you lose nothing at the tail end. Note it is `log(1 + x)`, not `log(x) + 1` — a surprisingly common misreading. **Square root, `sqrt(x)`** — defined for `x >= 0`, mild compared with the log. It is the traditional variance-stabilising choice for count data, where the variance grows roughly with the mean. **Box-Cox** — a one-parameter family for strictly positive x: ``` T(x) = (x^lambda - 1) / lambda for lambda != 0 T(x) = log(x) for lambda = 0 ``` lambda is not chosen by hand: it is estimated by maximum likelihood, picking the value that makes the transformed column look as close to normal as the family allows. lambda = 1 is essentially no transform, 0.5 is the square root, 0 is the log, -1 is the reciprocal. Because it needs `x > 0`, Box-Cox suits a strictly positive quantity such as insurance claim severity but not a column with zeros. **Yeo-Johnson** — the extension of Box-Cox to the whole real line. It applies a shifted power transform to values at or above zero and a mirrored power transform to negative values, so it is defined for every real input while keeping a single lambda estimated the same way. This is the transform for a `net profit` column that legitimately goes negative for loss-making accounts. ## The bookkeeping that people get wrong **Fitted versus fixed.** `log`, `log1p` and `sqrt` are fixed functions: no parameter is learned, so applying them to the whole dataset leaks nothing. Box-Cox and Yeo-Johnson learn a lambda from data. That lambda is a fitted parameter and must be estimated on the training fold only, then reused unchanged on validation, test and production rows. Estimating it on everything is quiet leakage. **Shifting to force positivity.** The tempting hack for zeros and negatives is `log(x + c)` for some constant c. It works arithmetically but the result depends entirely on c: a tiny c stretches the near-zero values into a huge artificial left tail, while a large c makes the transform nearly linear and pointless. If you must shift, justify c from the domain (a known measurement floor), not from what happens to look pretty. **Zeros that mean something else.** A point mass at zero is often a different state, not a small value — `never purchased` rather than `purchased a little`. Squeezing it into the same continuous scale hides that. The stronger design is often a pair of features: a binary `spent anything` flag plus the transform applied to the positive part. **Which models care.** Skew transforms of an input feature matter to linear and penalised linear models, distance-based methods and anything with a squared-error loss on that input. A tree ensemble splits on order alone, so any strictly increasing transform of a single feature leaves its achievable splits unchanged and buys nothing. Reaching for a power transform to fix a tree model's accuracy signals a missing mental model of how trees split. **Interpretability cost.** After a log-family transform, a linear coefficient is read per unit of the transformed scale — roughly a proportional effect rather than an absolute one. That is often an advantage for money-like quantities, but you have to say so out loud when you present coefficients. ## A decision order that holds up 1. What is the column's range? Strictly positive, non-negative with zeros, or genuinely signed? 2. Does the model care about shape at all? If it is a tree ensemble, stop. 3. Non-negative with zeros: try `log1p` first, `sqrt` if the squeeze is too aggressive. 4. Strictly positive and you want the data to pick the strength: Box-Cox, lambda fitted on train. 5. Signed values: Yeo-Johnson. 6. Zeros that encode a different state: split into a flag plus a transformed positive part.