How do you search a space where an RBF kernel has a gamma but a linear kernel has none?

## Rectangular spaces versus real ones Both grid and random search are usually described over a rectangle: every hyperparameter has a range, and every combination of ranges is legal. Real configuration spaces are frequently not rectangles. Some hyperparameters exist only when another hyperparameter takes a particular value. The canonical example is a kernel choice: a radial basis function kernel is governed by a width parameter, and a linear kernel has no such parameter at all. Other everyday examples of the same shape: - a penalty *type* (L1, L2, or a mixture) where a mixing ratio only exists for the mixture - a solver choice where only one of the solvers accepts a momentum-like setting - a resampling strategy where the sampling ratio only exists if resampling is enabled at all ## What the flat cross-product costs Take kernel in {linear, RBF}, five values of the penalty strength, five values of the kernel width. The rectangular grid is `2 * 5 * 5 = 50` cells. On the linear branch, the width does nothing, so those 25 cells collapse to five genuinely different models: the other 20 fits are exact repeats, and under 5-fold cross-validation that is 100 redundant fits. The waste is not merely compute. The trial log now contains 25 rows for the linear kernel and 25 for RBF, which makes the linear kernel look as though it were explored as thoroughly, when in truth it was explored five ways. Random search over the same flat space wastes in the same way, just stochastically: half the draws land on the linear branch and their width draws are noise recorded against a model that never saw them. ## The conditional (tree-structured) formulation Declare the space as a tree. The kernel is the parent node with two children. Under `linear`, the only free hyperparameter is the penalty strength. Under `rbf`, there are two: penalty strength and width. A draw walks the tree — pick the kernel, then draw only the parameters that branch owns. Nothing inactive is ever sampled, nothing duplicate is ever fit, and the trial log records only parameters that were actually in force. If your tooling will not express a conditional space, two workable fallbacks: 1. **Split the search.** Run one search per branch. This is often the clearest option because it forces you to state a budget per branch, and it makes the branch comparison explicit. 2. **Deduplicate.** Sample the flat space but canonicalise each configuration — set inactive parameters to a fixed sentinel value before hashing — and skip any configuration already evaluated. This recovers the compute but still needs care when you read the log. ## Budgeting across branches Branches do not deserve equal budgets. A branch with two free continuous hyperparameters needs more trials than a branch with one to reach comparable coverage; the rough rule is to scale trials with the number of live axes on the branch, not with the number of branches. If you split the search, say so explicitly: 40 trials for RBF and 20 for linear is a defensible split of a 60-trial budget, an even 30/30 usually is not. One caveat when comparing branches: the branch that got more trials also got more chances to look good, so a marginally higher best score on the larger branch is not by itself evidence that the branch is better. ## Discrete and integer axes in the same breath Conditional structure usually arrives alongside non-continuous axes, so the same answer should cover: - **Categorical parameters** have no ordering. There is nothing between `linear` and `rbf`, so they are drawn from a list, and imposing a numeric encoding on them so the space stays rectangular is a modelling error, not a convenience. - **Integer parameters** such as depth or a neighbour count are drawn from a discrete range, or drawn continuously and rounded *before* the fit. Log the rounded value; if the log records 7.4 while the model trained at 7, your record of the search is wrong, and duplicate rounded values should be detected rather than re-fit. - **Ordered but unevenly spaced choices** — a list of candidate leaf sizes, say — are best given explicitly as a list rather than a range, so the search does not pretend to a resolution the model cannot use. ## What an interviewer is checking That you have noticed real spaces are not rectangles, that you can quantify the duplication a flat cross-product causes, and that you know a per-branch budget is a decision you have to make rather than a detail the sampler handles for you.