Your TF-IDF matrix has 50,000 columns for 20,000 documents — how do you decide what to prune?

## Is 50,000 columns actually wide? The reflex is that having more columns than rows is dangerous. Before acting on that reflex, do the arithmetic. Stored densely, 20,000 x 50,000 is one billion cells — at eight bytes each, about eight gigabytes, most of it zeros. Stored sparsely, only the non-zero entries exist. A document containing a hundred distinct terms contributes a hundred entries, so the whole matrix is roughly two million values plus index overhead: a few tens of megabytes. The density is about 0.2%. So the first answer to "we have 50,000 columns" is: that is normal for text, it is cheap in the right storage format, and the column count on its own is not evidence of a problem. What *would* be evidence: - Training or scoring time is unacceptable, and profiling points at the feature matrix. - The serialised artefact — the vocabulary map plus the idf vector — is too big to ship to where it has to run. - Held-out performance improves when you prune, which is the only argument that settles it. ## The shape of a text vocabulary Term frequencies in natural language are extremely skewed. A small number of terms occur in nearly every document; an enormous tail occurs once or twice in the entire corpus. In a typical corpus a very large share of the distinct terms — often more than half — appear in exactly one document. Those are typos, product codes, one-off proper nouns, tokenisation debris. That shape is what makes pruning cheap. Cutting the tail removes most of the columns and almost none of the mass in the matrix, because those terms occur in one document each by construction. ## The dials **Minimum document frequency.** Drop any term appearing in fewer than `k` documents (or fewer than some fraction of them). `k = 2` alone often halves the vocabulary. The argument is statistical as well as economic: a term seen in one training document gives the model a column that can only ever fire for that document, which is an invitation to memorise. Raising `k` to 5 or 10 is common on large corpora. **Maximum document frequency.** Drop any term appearing in more than, say, 90% of documents. This is a corpus-derived stopword list — it removes whatever *this* corpus treats as filler, which beats a generic list because the filler is domain-specific. In a corpus of support tickets, `ticket`, `issue` and `customer` may be near-universal and useless even though no standard stopword list contains them. Note the interaction with idf: those terms already have weights close to zero, so removing them mostly buys memory and clarity rather than accuracy. **A hard cap on vocabulary size.** Keep the `V` most frequent terms overall. Blunter than the two cut-offs, but it gives a predictable artefact size, which matters when you have a memory budget to hit. **Explicit stopword removal.** Fine, but largely redundant once idf and a maximum-document-frequency cut-off are in play, and it introduces a hand-maintained list that drifts away from the data. ## Two things people get wrong **Pruning is fitted, so it lives inside the fold.** A document-frequency threshold decides which columns exist. Compute it over the whole corpus and held-out documents help create or destroy features, which biases your estimate. Fit it on the training portion of each fold along with the vocabulary and the idf values. **Do not confuse pruning with dimensionality reduction.** Cutting rare terms removes columns you decided are not worth their weight. Projecting the matrix into a smaller dense space is a different operation with different tradeoffs, and it destroys the one-column-one-term correspondence that makes a sparse text model debuggable. If the reason you wanted fewer columns was to keep the model explainable, pruning helps and projection hurts. ## How to actually decide Treat the cut-offs as hyperparameters, not as hygiene. 1. Establish the baseline with no pruning, so you know what you are trading against. 2. Sweep the minimum-document-frequency threshold across a few values and watch held-out score, artefact size and training time together. 3. Expect a plateau: score is usually flat over a wide range of `k` while vocabulary size falls dramatically. Take the aggressive end of the plateau. 4. Only if the plateau slopes downward do you have a genuine accuracy-versus-size tradeoff to escalate, and then it is a product decision about latency or memory budget, not a modelling one. The interviewer is listening for whether you reach for arithmetic before reaching for a knob, and whether you know that the answer is measured rather than asserted.