In target encoding, a seller with three historical rows all converted — smooth toward the prior or bucket it as rare?

## The problem both mechanisms solve Out-of-fold construction stops a row's own label from entering its own feature, but it does not make a mean computed from three rows trustworthy. Three conversions out of three gives `mean = 1.0`; the honest reading is "this seller might be anywhere from average to excellent, and we have almost no evidence." Handed to a tree, that 1.0 sits next to the 1.0 of a seller with 2,000 rows and a genuinely perfect record, and the model treats them as the same. Across 40,000 sellers, most of which have a handful of rows, this noise floods the column. ## Smoothing toward the prior The standard remedy is a weighted blend between the level's own mean and the global target mean: ``` enc_c = (n_c * mean_c + m * prior) / (n_c + m) ``` - `n_c` is the level's row count in the encoding data, `mean_c` its target mean, `prior` the overall target mean. - `m` is a **pseudo-count**: literally, the number of imaginary rows at the global rate that you add to every level. It is the amount of evidence a level must accumulate before its own mean starts to dominate. At `n_c = m` the encoded value sits exactly halfway between the level mean and the prior. As `n_c` grows the prior term becomes irrelevant; as `n_c` shrinks toward zero the encoding converges on the prior. This is empirical-Bayes shrinkage in its simplest form, and it is self-adjusting — small levels are pulled hard, large levels are left alone, with no threshold anywhere. `m` is a hyperparameter and should be tuned like one, on the same validation scheme as everything else. Small `m` gives a sharper but noisier feature; large `m` collapses the column toward a constant. A variant replaces the hard pseudo-count with a smooth weighting function of `n_c`, but the behaviour is the same and the tuned pseudo-count is easier to explain. ## Collapsing rare levels into one bucket The alternative is a hard cut: every level with fewer than, say, 100 rows becomes the single level `rare`, and that bucket gets its own encoded value — the target mean across all the rows it swallowed. On an app-store category column, dozens of niche categories with a handful of listings each collapse into one bucket with a stable, well-estimated rate. What you gain: a much smaller mapping to store and serve, a stable estimate, and a natural, already-modelled destination for a level the serving path has never encountered. What you lose: every distinction between rare levels, including a genuinely bad actor with 90 rows that would have survived smoothing. And the threshold is arbitrary — 100 is a habit, not a derivation; it should be checked against the count distribution and, if it matters, tuned. ## Choosing, and combining They are not exclusive, and in a real pipeline they usually stack: 1. Collapse levels below a small floor (levels with one or two rows, or the tail that accounts for a negligible share of traffic) into `rare`, mostly to bound the size and refresh cost of the mapping. 2. Smooth everything that survives, so the mid-tail levels degrade gracefully instead of falling off a cliff at the threshold. A useful third move is to keep the level's row count as its own feature alongside the encoded mean. The model can then learn "trust this rate more when the count is high", which recovers some of the information smoothing deliberately blurs — and the count is often independently predictive. ## How to tell it is working Look at the distribution of the encoded column. Spikes of mass exactly at 0.0 and 1.0 mean tiny levels are being trusted at face value. After smoothing, the histogram should be a hump around the prior with tails reaching out only where the counts justify it. Then check the score: if increasing `m` keeps improving validation, the column was noise-dominated; if performance falls off immediately, the level signal is strong and real and you can afford a lighter hand. ## The mistake to avoid Do not use smoothing as a substitute for out-of-fold construction. They fix different failures: shrinkage attacks the *variance* of an estimate from few rows, folds attack the *bias* of reading a row's own label. Heavy smoothing does mask the self-label leak on large levels — with `m = 100` the single-row seller no longer encodes to its own label — but it hides the mechanism rather than removing it, and the smallest levels are precisely where the contamination is largest. Use both.