Recursive feature elimination over 400 marketing-attribution columns runs for hours — what is it doing and how do you cut the cost?

## The loop Recursive feature elimination is defined by four steps repeated until a stopping size is reached: 1. Fit the model on the columns currently in play. 2. Ask the fitted model for a ranking of those columns — the magnitude of each fitted weight for a linear model, or whatever internal importance the learner exposes. 3. Remove the lowest-ranked column, or the lowest-ranked block of columns. 4. Go back to step 1 with the smaller set. It is a wrapper method: it uses the model to decide, and it pays for that in fits. It is also specifically a **backward** procedure, which matters, because backward elimination judges each column in the presence of the others still standing. ## Why it refits instead of ranking once The obvious cheaper thing is to fit once, take the ranking, and keep the top k. That is a perfectly legitimate method — but it is a different one, closer to a filter that happens to use model weights as its statistic. The reason to refit is that rankings **change** as columns leave. Two near-duplicate columns split their influence while both are present, so both look mediocre; remove one and the other's weight jumps, revealing that the information was valuable all along. A single-pass ranking would have dropped both. Every refit re-evaluates the survivors in the context that actually remains, and that context-dependence is the entire value of the recursive form. ## Where the hours go Count fits. - Removing **one column per round**, from 400 down to 20, is 380 rounds, so 380 fits. - If you also want the loop to choose the final size, you score each candidate size on cross-validation folds. At five folds that is roughly 1,900 fits. - If the model itself takes even ten seconds, 1,900 fits is more than five hours. Nothing pathological is happening; the arithmetic is just unforgiving. ## The levers, in order of payoff **Remove a block per round.** Dropping a fixed fraction — say 10% of the remaining columns each round — turns the 400-to-20 journey into roughly thirty rounds rather than 380, a better-than-tenfold saving. The cost is granularity: you can no longer distinguish the exact column that should have been the 37th to go. In practice that granularity is worth almost nothing at the wide end of the loop and quite a lot at the narrow end, which leads to the next lever. **Go coarse-to-fine.** Use large blocks while many columns remain and switch to single removals once the set is small. You spend your fits where the decisions are actually close. **Pre-screen with a filter.** A cheap univariate pass that takes 400 columns to 120 before the loop starts removes most of the work outright. This is the standard cascade and it is why filters and wrappers are complements rather than rivals. The risk is the filter's known blind spot — a column that only matters jointly with another can be cut before the wrapper ever sees it — so keep the pre-screen generous rather than aggressive. **Rank with a cheaper model.** Run the elimination using a fast learner to produce the rankings, then fit and validate the expensive model once on the resulting set. You are trading fidelity for time, and the trade is often good, because the ranking only needs to be roughly right about which columns are worst. ## Two correctness checks worth making First, **the ranking must mean something**. If you rank by coefficient magnitude, columns on wildly different scales produce coefficients on wildly different scales, and the elimination order will partly reflect units rather than usefulness. Put the columns on a common footing before you rank that way. Second, **the chosen set is tied to the ranking model**. What a linear learner considers dispensable is not what a tree-based learner considers dispensable. If the production model is not the model that drove the elimination, treat the result as a candidate set to be validated, not as a conclusion. ## Stability One more senior habit: the output of a single elimination run is one draw from a noisy process. Re-run the elimination on several bootstrap resamples of the rows and look at how often each column survives. Columns selected in most runs are a defensible feature set; columns selected in half the runs are a coin flip that happened to land your way, and the difference is invisible if you only ever run the loop once.