Why does a linear model usually beat boosted trees on sparse n-gram text features?

### The shape of bag-of-n-gram text Represent documents as counts or weights over unigrams and bigrams and you get a matrix that is wide, sparse and additive. Wide: 50,000 to several hundred thousand columns is ordinary. Sparse: a single review touches a few dozen of those columns, so well over 99% of the matrix is zero. Additive: the evidence that a review should be routed to billing rather than shipping is spread across hundreds of individually weak cues - refund, charged, invoice, double, statement - none of which decides the case alone. Model choice on text is really a question about that shape, not about the words. ### Why the linear family fits the shape A linear model is exactly this shape written down. One weight per term, a document's score is the sum of the weights of the terms it contains, and every one of the hundreds of weak cues contributes in a single pass. Regularization does the work that would otherwise be impossible in 50,000 dimensions: an L2 penalty spreads credit across correlated near-synonyms, an L1 penalty prunes the vocabulary to the terms that carry weight. Regularized logistic regression and linear support vector machines have been the strong, hard-to-beat baselines for sparse text classification for decades, and they train in seconds on millions of documents because the arithmetic touches only the non-zero entries. ### Why a boosted tree ensemble fights the shape Three mechanisms work against trees here. - One feature per split. A split tests a single column - does the term refund appear at least once. To accumulate the hundreds of weak cues a linear model sums in one step, the ensemble needs a separate branch, in a separate tree, for each of them. Thousands of trees can approximate what one weight vector encodes directly, at far greater cost. - Rare terms give unbalanced splits with almost no gain. A term appearing in 0.3% of documents sends 99.7% of the rows down one branch. The impurity improvement is tiny, so the greedy search often prefers a frequent, uninformative term, and the informative rare cues are never used. - The split search itself is expensive and noisy over that many candidate columns, and the interactions trees exist to capture - this feature matters only when that one is high - are not how lexical evidence usually behaves. Trees are being asked to pay for a capability the data does not need. None of this makes boosting impossible on text; with heavy feature selection and enough tuning it can be pushed to a respectable score. The claim is about equal effort: at the same budget, a regularized linear model normally wins on raw sparse n-grams, and it gets there far faster. ### When the tree family comes back Change the representation and the argument disappears. Compress each document into a few hundred dense continuous features - a topic or similarity score, a length, a readability number, the output score of a linear text model - and you have an ordinary tabular problem where boosted trees are back in their element. That hybrid is the usual production shape for a routing system: a linear model reads the text and emits one score, and a tabular model combines that score with metadata such as account age, order value, channel and prior contacts, where genuine interactions between those columns do exist. Feeding raw n-gram columns and a dozen metadata columns into the same tree ensemble is the worst of both worlds - the metadata signal is buried among 50,000 near-empty candidate splits. ### Diagnosing rather than assuming The way to know the linear choice is right is to look at where it fails. If errors concentrate on cases where meaning depends on two tokens co-occurring - a negation next to a positive word, a product name next to a complaint verb - additive per-term weights genuinely cannot express that, and the cheap fix is to add the interacting phrases as explicit terms rather than to change family. If errors are spread evenly and the confusion is between genuinely similar categories, more capacity is not the problem; the labels or the taxonomy are.