Feature Engineering and Data Preparation

Choose the entity and a reference date, keep only that entity's events inside a window ending at the reference date, then collapse them into one row: counts, sums, means, maxima, distinct counts, and days since the last event.

Impute in most cases. Deleting the rows discards a fifth of the data and has no equivalent at scoring time, where every request must return a prediction. Drop the column only if it is nearly empty or adds no lift.

Education level has a real order - high-school below bachelor below master below PhD - so integer codes carry that information. Job family has no order, so codes 0-3 invent a ranking the model takes literally. Encode unordered categories as one-hot indicators instead.

Z-score standardisation subtracts the mean and divides by the standard deviation, producing mean 0, standard deviation 1, and no fixed bounds. Min-max scaling rescales values into a fixed range such as 0 to 1 using the training minimum and maximum.

Equal-width bins cut the range into intervals of the same size, so a long tail crowds almost every row into the first bin and leaves near-empty bins above. Equal-frequency bins cut at quantiles, so counts are similar and widths vary.

Random oversampling duplicates existing minority rows until the classes are closer to balanced. Random undersampling deletes majority rows to reach the same ratio. Oversampling keeps every row but repeats information; undersampling throws information away.

TF-IDF multiplies each word's count by an inverse-document-frequency factor that shrinks as the word appears in more documents. Words like 'the' and 'and' sit in nearly every review, so their columns collapse to near-zero while rarer words keep large weights.

An information value of 0.015 sits below the conventional 0.02 floor, so on its own the feature barely separates defaulters from non-defaulters. Standard practice is to drop it from the scorecard unless policy requires it or it earns its place alongside other characteristics.

A class weight multiplies every loss term from that class by a constant, so each rare positive counts like many rows. The optimiser trades more errors on the common class for fewer on the rare one, and fitted probabilities rise.

A quotient is not a linear combination of its two columns, so no coefficients on debt and income reproduce debt divided by income. A tree can only approximate it with a staircase of axis-aligned splits, which costs depth and data.

Filters rank columns with a cheap statistic and never train the model. Wrappers train it many times on candidate subsets and keep the best. Embedded selection falls out of one model fit. Cost and model-specificity rise across the three.

Computing each seller's target mean over all training rows puts a row's own label inside its own feature, so the model reads the answer back. Out-of-fold encoding builds each fold's means from the other folds only.

Least squares squares residuals, so a far-out point dominates the total and rotates the line. kNN puts that huge gap into every distance, so neighbourhoods break. A tree splits on rank order, so the extreme lands in its own branch.

kNN and SVM compare examples by distance, so a feature measured in large units dominates every comparison. A tree splits one feature at a time at a threshold; rescaling preserves the ordering of values, so exactly the same splits remain available.

Plain log fails here: log(0) is undefined and negatives have no log. For a non-negative column with zeros, use log(1+x). When values drop below zero, use Yeo-Johnson, the power transform defined on the whole real line.

SMOTE picks a minority row, finds its k nearest neighbours among other minority rows, chooses one of them at random, and places a new point at a random fraction of the way along the straight line between the two.

The vocabulary and the idf values are learned parameters. Building them over the whole corpus before splitting lets held-out documents shape the features that describe them, so validation scores come out optimistically biased and overstate what production will do.

Weight of evidence replaces a bin with ln(share of non-defaults / share of defaults), which is the bin's log-odds offset from the population. That makes every feature monotone, on one common scale, linear in log-odds, auditable, and lets missing values form their own bin.

If a share of labels are wrong, even a model that predicts the true answer every time scores below 100% against those labels. Two radiologists disagreeing on one scan set a floor under any model's error.

Ask whether B2B orders will be scored in production. If so, keep the row and represent that segment explicitly; deleting it trains a model blind to your most valuable orders. Drop only corrupt or out-of-scope rows.

Count encoding replaces each ZIP code with how many rows carry it, turning 30,000 levels into one numeric column. It helps when frequency proxies something real, like population density. It uses no labels, so it cannot leak.

Duplicated rows count the same information several times. They upweight those examples in the loss, inflate one class's apparent frequency, and make the real sample size much smaller than the row count suggests.

-999 is a code the logger writes when the sensor is disconnected, not a measurement. Capping folds it into the real data range and teaches a relationship that never existed. Convert it to missing instead.

Window length trades freshness against stability, and the right value depends on how fast the behaviour changes. A 7-day window reacts quickly but is sparse and noisy; a 90-day window is stable but dilutes recent change. Tune it on validation data rather than guessing.

A correlation or mutual-information filter scores each column against the target on its own. Two near-duplicates both score well and survive; a feature that matters only alongside another scores near zero alone and is cut. The blind spot is univariate scoring.

Because filling destroys the fact that the value was absent, and absence is often predictive in its own right. A binary flag restores that information, and it lets the model treat filled rows differently from genuinely observed ones.

Percent agreement counts the agreements two annotators would hit by chance alone. Cohen's kappa removes that baseline: kappa = (observed agreement - chance agreement) / (1 - chance agreement), so 0 means chance-level and 1 means perfect.

It adds fifty binary columns that are 98 percent zeros and gives every state its own weight. Each weight is driven only by that state's rows, so small states get unstable estimates and total variance rises. Penalising the weights becomes close to mandatory.

Percentile caps are parameters estimated from data. Computing them over every row lets held-out rows shape their own preprocessing, so validation scores turn optimistic. Fit the bounds on training rows and apply those stored numbers unchanged everywhere else.