How do skip-gram and CBOW differ, and which wins on a small, rare-word-heavy corpus?

## The shared setup Both objectives come from the same family of shallow predictive word-vector models, and both rest on the distributional hypothesis: words that occur in similar contexts should get similar vectors. Training data is created mechanically from raw text with no labels. A window slides over the corpus; at each position one token is the *centre* word and the tokens within the window on either side are its *context* words. The model itself is deliberately tiny — there is no hidden non-linearity. There are two parameter matrices, each with one row per vocabulary word: an *input* (centre) matrix and an *output* (context) matrix. A score for a (centre, context) pair is the dot product of the centre word's input vector and the context word's output vector. Training pushes the score of observed pairs up and the score of non-observed pairs down. When training finishes, the input matrix is normally kept as "the word vectors" and the output matrix is discarded (summing or concatenating the two is an occasionally used variant). ## The two directions **CBOW (continuous bag of words)** takes the context and predicts the centre. The input vectors of all context words in the window are averaged into one vector, and that average is scored against the output vectors to predict which word sits in the middle. "Bag of words" is literal: the average throws away order, so `the cat sat on` and `on sat cat the` give the same input. **Skip-gram** reverses this. It takes the centre word's input vector and predicts each context word independently. One window of half-width 5 with 10 neighbours becomes 10 separate (centre, context) training pairs, each with its own gradient step. ## Why the direction changes the outcome *Updates per occurrence.* A word that appears 40 times in the corpus receives roughly 40 centre-word updates under CBOW and roughly 40 x (window size) targeted updates under skip-gram. For rare words that difference is the whole game — skip-gram simply squeezes more supervision out of every appearance. *Averaging blurs.* Under CBOW, a rare word appearing as *context* is one of several vectors going into an average, and the average is dominated by whatever frequent words share the window. Its individual contribution to the gradient is diluted. Skip-gram never averages, so each pair is scored on its own. *Cost.* CBOW does one forward/backward pass per window versus skip-gram's several, so CBOW trains several times faster on the same corpus. On very large corpora that speed can buy more epochs or a larger vocabulary, and frequent words have ample data either way, so the quality gap narrows. The original reports are consistent with this: CBOW is faster and slightly better on syntactic regularities for frequent words; skip-gram is better for rare words and small data. ## The window is not just a width Two details of the window matter more than candidates expect. First, the window is typically *sampled*, not fixed: for each centre word an actual half-width is drawn uniformly between 1 and the configured maximum. Words adjacent to the centre therefore fall inside the window in every draw, while words at the far edge only make it occasionally. The effect is a soft distance weighting for free, plus a cheaper average cost. Second, window size changes *what kind of similarity* you get. Narrow windows (1-2) capture words that are grammatically substitutable — nearest neighbours look like the same part of speech and the same syntactic slot. Wide windows (10+) capture topical relatedness — nearest neighbours are words from the same subject area rather than words you could swap in. Neither is more correct; you pick to match the downstream use. A retrieval-style feature usually wants the wide, topical setting; a tagging feature usually wants the narrow one. A third pre-processing step, subsampling of very frequent words, discards common function words with a probability tied to their frequency. This both speeds training and effectively widens the window, since discarded tokens no longer occupy window slots. ## Choosing in practice Ask two questions. How much text do I have, and how much of the vocabulary I care about is rare? A domain corpus of a few million tokens where the interesting vocabulary is long-tail technical terms points squarely at skip-gram. A very large generic corpus where the goal is decent vectors quickly points at CBOW. If the rare vocabulary is not just rare but *absent* at inference time, neither objective helps by itself — that is a vocabulary-coverage problem and needs subword-composed vectors instead.