How do you choose among XGBoost, LightGBM and CatBoost for a given table?

## They are variants of one algorithm XGBoost, LightGBM and CatBoost are all gradient boosted decision trees: an additive ensemble where each new tree is fitted to the negative gradient of the loss with respect to the current predictions and added with a shrinkage factor. They differ in how they find splits, how they grow trees, how they treat categorical columns and how their objective is regularized. Those differences change *cost and convenience* far more often than they change *accuracy*. That framing matters in an interview, because the weak answer is a ranking ("LightGBM is the best") and the strong answer is a decision procedure. ## What each one is actually optimised for **XGBoost** — a regularized objective that uses second-order (curvature) information about the loss when scoring candidate splits, with explicit L1 and L2 penalties on leaf weights and a learned default direction for missing values at each split. Grows trees level by level by default. It is the conservative general-purpose choice: predictable, heavily regularized, widely operated. **LightGBM** — designed above all for throughput and memory on large data: continuous features are bucketed into a small number of bins so split search scans histograms rather than sorted values, and trees are grown best-first rather than level by level. The practical consequence is that it trains dramatically faster on wide and tall tables. The practical caution is that its growth strategy will happily build very unbalanced trees, so on small data you must cap tree size and require a minimum number of rows per leaf, or it overfits. **CatBoost** — designed around categorical data: it converts categorical levels into numeric statistics computed in a way that avoids letting a row's own target leak into its own feature value, so you do not have to hand-build an encoding. Its trees are symmetric — every node at a given depth uses the same split — which makes prediction unusually fast and uniform in cost. It also tends to be strong at or near its defaults. Deliberately shallow treatment of the internals here: what matters for the choice is what each design *buys you*. ## The decision procedure **1. Start from the constraints, not the leaderboard.** - *Scale and retrain window.* A **60-million-row, 2,000-column table with a nightly retrain window** is a throughput problem before it is an accuracy problem. Binning-based split search and fast parallel histogram building are what make that window achievable, which points at LightGBM. If the model cannot be rebuilt before the batch job that consumes it, its error rate is irrelevant. - *Categorical cardinality.* A **marketplace table dominated by three categorical columns with tens of thousands of levels each** — seller id, product category, postcode — is the case for CatBoost. One-hot encoding explodes the width; hand-rolled target encoding is fine in principle but leaks unless it is computed strictly inside each training fold, and that is a bug people ship constantly. - *Inference budget.* Symmetric trees evaluate at uniform depth with very predictable cost, which matters when you serve per-request rather than in batch. - *Neither dominant.* Take XGBoost as the default and spend the saved time on features and validation. **2. Run a fair bake-off.** Same folds, same metric, same early-stopping rule against the same held-out data, and *comparable* tuning budgets. The single most common way this comparison goes wrong is spending a day tuning your favourite and running the others at defaults — that measures your effort, not the library. **3. Expect a small gap.** On most business tables the tuned results sit within fold-to-fold variance of each other. When that happens, say so and decide on the things that are not within noise: training time, memory, inference latency, reproducibility across versions, and whether the team can debug it at 3am. ## What the choice does not fix None of the three rescues a bad target definition, leakage from a feature computed after the event you are predicting, or a validation split that ignores time or grouping. Candidates who reach for a different library when a model underperforms are usually solving the wrong problem: the leakage audit and the validation protocol come first, and only then does it matter which booster you fit. ## The answer to give "They are the same algorithm with different engineering priorities. I pick on the binding constraint — throughput, categorical cardinality or inference latency — default to the regularized general-purpose option when nothing binds, and then prove the choice with a like-for-like comparison instead of asserting a ranking."