Why does a static word vector give 'bank' one vector for both of its meanings?

## One row per spelling A static word-vector model is, at inference time, a table: a word type maps to a fixed vector. There are as many rows as there are word types in the vocabulary, and the row is selected by the surface form alone. Nothing about the surrounding sentence is consulted, because there is no mechanism to consult it — the vector was fixed when training ended. During training, every occurrence of the spelling `bank` — in `sat on the river bank`, in `the bank raised interest rates`, in `bank of fog` — produces gradients on the *same* row. The objective has no notion of sense; it sees a token and looks up an index. So the row is pulled toward whatever makes it predictive of *all* those contexts at once. ## What the compromise actually looks like The result is not a vector that cleanly represents either sense, and it is not a vector that splits into two clusters given enough epochs — there is only one row, so there is nothing to split. It settles at a frequency-weighted compromise: roughly, the direction that minimises total loss across both context distributions, which sits between the two neighbourhoods and is a poor member of either. Two practical consequences follow. *Sense dominance.* If 95% of the corpus's `bank` tokens are financial, the vector is essentially the financial sense with a slight pull toward the geographic one. Retrieval or classification features built on it will behave as if the rarer sense does not exist. Change corpora — a hydrology corpus instead of a news corpus — and the same word's vector moves substantially. *Polysemy is common, not exotic.* This is not a corner case affecting a handful of puns. High-frequency words are the most polysemous ones in most languages, so the words your model sees most often are exactly the ones whose vectors are most blurred. ## The related surprise: antonyms are neighbours The same distributional logic produces an effect that trips up candidates who describe these vectors as "capturing meaning". `hot` and `cold`, `increase` and `decrease`, `always` and `never` come out as close neighbours, because they occur in near-identical contexts — anywhere one is grammatical, the other usually is too. Cosine similarity in this space measures *contextual substitutability*, not semantic agreement. If your downstream task is sentiment or negation-sensitive, that is a genuine hazard rather than a curiosity. ## What analogy arithmetic does and does not prove The famous demonstration is `king - man + woman`, whose nearest neighbour is `queen`. It is a striking result, and it does show that some relational structure is encoded as roughly consistent offset directions in the space. But it proves less than it appears to. The standard evaluation protocol *excludes the three query words* from the candidate set before taking the nearest neighbour — and `king` itself is usually the true nearest vector to the query point. Without that exclusion the analogy frequently "fails". Reported analogy accuracy also varies sharply by relation type: geographic and morphological relations score far better than abstract semantic ones, and the aggregate number hides that. Treat an analogy score as one weak, coarse diagnostic of the space, never as evidence that the model has learned relational reasoning — and never as a substitute for evaluating on your actual downstream task. ## Living with it, or leaving it If you are stuck with static vectors, the mitigations are all partial: train on a domain corpus so the dominant sense is the one you care about; disambiguate upstream with a rule or a tag when a critical few words are the problem; or accept that features derived from polysemous high-frequency words will be noisy and lean on other features. The honest answer at interview, though, is that this limitation is structural. One vector per type is the defining property of the approach, and no amount of tuning removes it. It is the specific problem that motivated moving to representations that produce a different vector for each occurrence of a word, and being able to name that progression — and the reason for it — is what the question is really testing.