Cross-Validation and Model Selection

The aggregate summarises the entire timeline, so a row from year one already carries year three's demand. Every training row sees the future and every validation row sees its own outcomes, no matter how correctly the folds are ordered.

Prefer Hyperband when a cheap fidelity ranks configurations like full training and workers are plentiful; prefer Bayesian search when evaluations are expensive and few. Hybrids sample configurations with a surrogate and schedule them with brackets.

Start from the compute you have, not the size of the space. Fix the hyperparameters that rarely matter at defaults and give the whole budget to the two or three that plausibly move the score.

Skip nesting when a single large tuning split is already precise — millions of rows, few candidates, a decision with a wide margin. Keep it when data is scarce or wide and many settings are compared, where the optimism is largest.

That hold-out is no longer a test set: forty scored comparisons made it a validation set, and its number is optimistic by an unmeasurable amount. Get fresh untouched rows for the real verdict, and budget future use.

Turn the continuous target into a temporary discrete label — quantile bins such as quartiles — and stratify the folds on that bin. Every fold then carries a similar spread of target values, steadying fold metrics on small or skewed data.

Warm-start by seeding the surrogate with past trials, not by trusting last week's winner. Old scores do not transfer to new data, so re-evaluate the incumbent, keep some random exploration, and cold-start periodically to catch a moved optimum.

Each bootstrap replicate trains on only about 63% distinct rows, so its models are weaker and the error on left-out rows comes out too high. The .632 estimator blends that pessimistic error with the optimistic training error, 0.632 to 0.368.

Treat the space as conditional rather than rectangular: sample the kernel first, and only draw the width parameter when the kernel that uses it was chosen. A flat cross-product instead fits the same linear model many times over.

Treat that swing as a measurement of your own noise floor. Repeat the whole cross-validation with several different random partitions and average across all runs, and refuse to call any model difference smaller than the swing you just observed.

No. Nested cross-validation estimates a tune-then-fit procedure, not one fixed setting, so outer folds are free to disagree. Disagreement usually means the score is flat across nearby candidates or the inner estimates are noisy.

A gap smaller than one standard error is indistinguishable from fold noise, so the two are tied on evidence. Among tied candidates the simpler one usually varies less on new data and is cheaper to run, explain and maintain.

Usually yes. Once the configuration is frozen, the validation rows are just labelled data, and training on more rows generally gives a better model. Report the test score for the refit model, and never redo selection after folding validation in.

A defensible option, not the default. Averaging the fold models gives a variance-reduced ensemble at no extra training cost, but costs k times the inference work, leaves no single auditable artifact, and was never scored as a unit.

Treat every mutable column as leaked until proven otherwise. A nightly rebuild overwrites fields with post-outcome values, so train only on columns that cannot change after the decision, or reconstruct earlier values from an append-only change history.

Start from what the model faces at inference. Break the axis production does not repeat: hold out unseen customers if it scores strangers, hold out later time if it re-scores known customers, and hold out both when it does both.