Filter, Wrapper, Embedded

## The three families Every feature-selection method answers the same question — which columns do we keep — but they differ in what evidence they use and how much computation they spend getting it. The standard taxonomy has three families. **Filters** score columns using a statistic computed from the data alone, before and independently of any model. Typical scores are the absolute correlation between a column and the target, an estimate of mutual information between a column and the target, a difference-of-means style test statistic across classes, or an unsupervised hygiene check such as a variance threshold. You compute one number per column, sort, and keep the top k or everything above a cutoff. **Wrappers** treat the learner as a black box and search the space of subsets, scoring each candidate subset by training the model on it and measuring held-out performance. Forward stepwise selection starts from nothing and repeatedly adds the column that most improves the score. Backward elimination starts from everything and repeatedly removes the least useful. Recursive feature elimination fits the model, ranks columns by the model's own weights, drops the weakest, and repeats. **Embedded** methods fuse selection into a single model fit: the training procedure itself decides which columns end up carrying influence, so you get a selected set as a side effect of fitting once. The specific mechanisms — a sparsity-inducing penalty on a linear model, or a tree learner's own choice of which columns to split on — are each covered under their own topics; what matters for the taxonomy is the interface, which is that you pay for one fit and receive a subset. ## Cost, in the units that matter Count model fits, because that is what dominates. - A filter costs **zero fits**. It is `p` cheap statistics for `p` columns, one sweep of the data. A 20,000-column table is not a problem. - A wrapper costs **many fits**. Forward selection over `p` candidates that stops after `k` steps evaluates roughly `p + (p-1) + ... ` candidates, on the order of `p * k` fits — and multiply that by the number of cross-validation folds used to score each candidate. Sixty candidate columns and four steps is already several hundred fits per fold. - An embedded method costs **one fit**, plus whatever hyperparameter tuning that model needed anyway. The practical consequence is that wrappers are affordable when `p` is in the dozens and the model trains in seconds, and infeasible when `p` is in the thousands and the model trains in minutes. ## What each family can and cannot see The cost ordering mirrors an information ordering. A univariate filter looks at one column at a time against the target. It therefore cannot notice that two columns say the same thing, and it cannot notice that a column is worthless alone but decisive in combination with another. It is also agnostic about the downstream model, which is a genuine virtue when you do not yet know what you will train, and a genuine limitation when the model has strong preferences. A wrapper sees exactly what the model sees, including interactions and redundancy, because the evidence is the model's own held-out performance on that subset. That fidelity is why it is the most accurate family in principle. It comes with two costs beyond compute: the chosen subset is tuned to one learner and does not necessarily transfer, and searching many subsets and reporting the winner produces an optimistically biased score unless the whole procedure is judged on data the search never touched. An embedded method also sees columns jointly, since the fit considers them together, but only through the lens of that one model's inductive bias. A subset chosen by one model family is a statement about that family, not a universal ranking of feature value. ## How they are actually combined The families are not competitors so much as stages. With 20,000 gene-expression columns and 80 samples, no wrapper is going to run and no wrapper's result would be stable at that sample size; a variance pass and then a univariate filter down to a few hundred columns makes the problem tractable, and something joint runs afterwards. With 60 sensor channels and plenty of rows, the filter stage buys almost nothing and the wrapper is affordable, so you skip straight to it. The question an interviewer is really probing is whether you pick a family from the shape of the problem — how wide the table is, how expensive one fit is, how many rows you have, and whether the selected set must be defensible to someone who did not choose the model.