In a bag-of-words matrix of movie reviews, what does TF-IDF weighting do to words like 'the' and 'and'?
answer
- raw counts reward the commonest words
- two factors multiplied per cell
- one factor is corpus-wide, not per document
- log of documents over document frequency
- a word in every review scores zero
basics
~20 sTF-IDF multiplies each word's count by an inverse-document-frequency factor that shrinks as the word appears in more documents. Words like 'the' and 'and' sit in nearly every review, so their columns collapse to near-zero while rarer words keep large weights.
solid answer
~50 sA bag-of-words matrix gives one row per review and one column per vocabulary word, holding raw counts. Under raw counts the biggest numbers belong to `the`, `and`, `of` — they are frequent in every review, so they carry almost no signal about whether a review is positive. TF-IDF rescales each cell to `tf * idf`, where `idf = log(N / df)`: `N` is the number of documents and `df` is how many documents contain the word. A word in every document has `df = N`, so `log(1) = 0` and its weight goes to zero; a word in a handful of documents gets a large multiplier. The result is that the largest entries in each row become the words that distinguish that review from the rest of the corpus, which is exactly what a classifier needs. It is a reweighting only — the vocabulary itself is unchanged.
code
python · 19 linesimport math
from collections import Counter
docs = ["the cat sat", "the dog sat", "the cat ran fast"]
tokens = [d.split() for d in docs]
n_docs = len(tokens)
df = Counter(term for doc in tokens for term in set(doc))
def tfidf(doc):
counts = Counter(doc)
return {t: (c / len(doc)) * math.log(n_docs / df[t]) for t, c in counts.items()}
for text, doc in zip(docs, tokens):
row = tfidf(doc)
print(text, "->", {t: round(w, 3) for t, w in sorted(row.items())})
# the cat sat -> {'cat': 0.135, 'sat': 0.135, 'the': 0.0}
# the dog sat -> {'dog': 0.366, 'sat': 0.135, 'the': 0.0}
# the cat ran fast -> {'cat': 0.101, 'fast': 0.275, 'ran': 0.275, 'the': 0.0}go deeper
Be ready to define bag-of-words in one sentence, write idf = log(N / df), and say out loud what happens to a word that appears in every document. That single example carries most of the answer.
Explain the two factors as separate objects: tf is per cell, idf is one number per term fitted over the whole corpus. Know the common tf variants and why row normalisation is applied.
Show you know what the representation costs in production: a dead column still costs memory, idf is a fitted parameter you must version and ship with the model, and word order is simply gone.
Frame it as a build-versus-buy call. Argue when a sparse count representation is still the right default for a text feature — cheap, inspectable, debuggable — and what evidence would justify moving to something heavier.
## The matrix you are building A text column cannot be fed to a classical model directly, so you turn a corpus of documents into a numeric matrix. **Bag-of-words** does it in the crudest possible way: fix a vocabulary of `V` distinct terms, give each document one row and each term one column, and put in cell `(d, t)` how many times term `t` occurs in document `d`. Twenty thousand movie reviews with a 50,000-word vocabulary become a 20,000 x 50,000 matrix. The name says what is thrown away: the document is treated as a *bag* of terms with no order, so "the film was not good" and "the film was good, not" produce identical rows. Most cells are zero — a single review touches only a hundred or so of the 50,000 columns — so the matrix is stored sparsely, as a list of the non-zero positions and values. ## Why raw counts mislead Natural language is dominated by function words. In a corpus of movie reviews, `the`, `and`, `a`, `of`, `to` are the highest-count terms in essentially every document. If you feed raw counts to a model, those columns have by far the largest values and the largest variance, yet they are almost identical across positive and negative reviews. They are volume, not signal. Meanwhile `masterpiece`, `tedious` or `refund` occur once or twice and would be numerically drowned out. ## What TF-IDF actually computes TF-IDF is a product of two factors, computed per cell. **Term frequency (tf)** is the within-document part: how much of *this* document is this term. Common choices are the raw count, the count divided by the document length, the binary flag "present or not", or the sublinear form `1 + log(count)`, which stops a term repeated twenty times from counting twenty times as much as a term appearing once. **Inverse document frequency (idf)** is the across-corpus part, one number per term, not per cell. With `N` documents and `df(t)` documents containing term `t`: ``` idf(t) = log(N / df(t)) ``` A term in every document has `df(t) = N`, so `idf = log(1) = 0`. A term in 1% of documents gets `log(100) ~ 4.6`. The factor rises as the term gets rarer, and it rises *logarithmically*, so a term in 10 documents is not 100 times more valuable than a term in 1,000 — just a couple of times. The cell value is `tf(d, t) * idf(t)`. So `the`, appearing in all 20,000 reviews, is multiplied by zero and its whole column becomes dead weight. `and` is nearly the same. `tedious`, appearing in 300 reviews, keeps a healthy multiplier and now dominates the row of any review that uses it. Many implementations use a smoothed variant such as `log((1 + N) / (1 + df(t))) + 1`, which keeps universal terms at a small positive weight instead of exactly zero and avoids dividing by zero for a term absent from the fitting corpus. The behaviour is the same in direction; only the floor differs. ## Length normalisation A long review has more of every word, so its whole row is inflated relative to a two-line review. Rescaling each row to unit length (dividing every entry by the square root of the sum of its squared entries) removes that, so what the model sees is the *composition* of a document rather than its size. This is usually applied after the tf-idf product. ## What idf does not do - **It does not delete anything.** The vocabulary is decided by the counting step. A zero-weighted `the` column still exists, still costs a column, and still has to be carried around. Removing terms is a separate decision (a stopword list, or a document-frequency cut-off). - **It is not a per-document statistic.** `idf` is fitted once over a corpus and then applied to every document, including future ones. That makes it a *learned* parameter, with all the consequences that has for train/test discipline. - **It does not know meaning.** `good` and `excellent` are unrelated columns. `not good` is invisible unless you also extract two-word n-grams. - **It does not fix class imbalance or rescale the target.** It is a feature transform, nothing more. ## The rule of thumb If a term is spread evenly across the corpus, it cannot discriminate between documents, and idf is the arithmetic that says so. That is the whole idea, and being able to state it in one sentence — plus the fact that `df = N` gives weight zero — is what an interviewer is listening for.
- What weight does a term get if it appears in every document of the corpus?Under plain `idf = log(N / df)` it is exactly zero, because `df = N` gives `log(1) = 0`, so the column is numerically inert. Smoothed variants like `log((1 + N) / (1 + df)) + 1` floor it at a small positive value instead, which keeps the column alive but tiny. Either way it stops competing with informative terms.
- Does TF-IDF change which words the model sees, or only how much they count?Only how much they count. The vocabulary is fixed by the counting step; idf is a per-term multiplier applied afterwards. A zero-weighted term still occupies a column and still costs memory. If you want it gone you have to drop it explicitly, with a stopword list or a document-frequency cut-off.
- Why is each TF-IDF row often rescaled to unit length?Because a long document has more of every term, so its raw row is uniformly larger than a short document's. Rescaling each row by its own magnitude makes the features describe the composition of the document rather than its size, so a five-line review and a five-paragraph review are comparable.
In a crowd where everyone is shouting, the words everyone shouts tell you nothing. TF-IDF turns down the volume on whatever every document says and turns it up on what only a few say.
saying these in an interview costs you the question
- Says TF-IDF removes stopwords from the matrix
- Computes idf per document instead of once over the corpus
- Claims TF-IDF captures word meaning or word order
- Assumes the highest raw count is the most informative word
- Thinks idf grows linearly as a term gets rarer