How do WordPiece and Unigram pick vocabulary differently from BPE's merges?

## Three selection criteria, one output shape Every subword tokenizer produces the same kind of artefact: a fixed vocabulary of pieces over a base alphabet, plus a rule for segmenting new text. The interesting differences are in **how a piece earns its slot** and **how segmentation is decided at encoding time**. ## BPE: greedy on frequency BPE grows the vocabulary bottom-up, merging the most frequent adjacent pair at each step and recording an ordered merge table. Encoding replays those merges in rank order. The criterion is raw count, with no model of the corpus behind it. The weakness this exposes: a pair can be frequent simply because both of its halves are extremely common, even though gluing them together explains nothing new. Merging `th` + `e` is unsurprising and adds little, yet it wins on count alone. ## WordPiece: greedy on likelihood gain WordPiece keeps the bottom-up loop but changes the score. It treats the corpus as generated by a unigram model over the current vocabulary and asks which merge most increases the likelihood of the data. Working through the algebra, the score reduces to the pair's frequency divided by the product of its two parts' frequencies — a mutual-information-flavoured quantity rather than a raw count. The practical effect is that WordPiece prefers pairs that *co-occur more than chance would predict*. A pair of pieces that are individually rare but almost always appear together scores highly; a pair of ubiquitous pieces that happen to land next to each other often does not. Vocabularies come out slightly more morpheme-like on the margin. WordPiece marks continuation pieces explicitly — the convention is a `##` prefix on any piece that is not word-initial, so `playing` might segment as `play` + `##ing`. Encoding is a greedy longest-match scan within each word against the vocabulary, not a merge replay. ## Unigram: top-down pruning under a probability model Unigram inverts the direction. It starts with a deliberately oversized candidate vocabulary — often generated by frequent substrings or a permissive BPE run — and fits a unigram probability distribution over those candidates using expectation-maximisation, where the hidden variable is the segmentation of each word. Then it computes, for each candidate token, how much corpus log-likelihood would be lost if that token were removed, and prunes a fraction of the least useful ones. Refit, prune, repeat, until the vocabulary hits the target size. Because it ends with an explicit probability for every token, encoding is a search rather than a replay: Viterbi over the lattice of possible segmentations to find the most probable one. That has a distinctive consequence — the tokenizer can also *sample* from the segmentation distribution instead of always taking the argmax. Training a model on sampled segmentations of the same word is subword regularisation, which makes downstream models more robust to how text happens to be split. BPE has an analogous trick in dropping merges at random during encoding, but it is bolted on rather than intrinsic. ## SentencePiece: the orthogonal axis SentencePiece is frequently listed as a fourth algorithm; it is not. It is an implementation that can train BPE or Unigram, and its distinctive contribution is a different treatment of the input. Rather than requiring language-specific pre-tokenization on whitespace, it consumes the raw character stream and encodes spaces as an ordinary symbol — conventionally the marker `▁` (U+2581) attached to the token that follows a space. Two things fall out. First, tokenization becomes language-agnostic: Japanese, Chinese and Thai have no whitespace to split on, and SentencePiece does not need any. Second, detokenization is exact and trivial — concatenate the pieces and turn the markers back into spaces — with no heuristics about where to reinsert whitespace. ## Choosing between them In practice the choice is driven less by measured quality gaps, which are modest, than by ecosystem and requirements: - **BPE (usually byte-level)** dominates modern generative models: simple, fast, deterministic, and byte-level variants give guaranteed coverage. - **WordPiece** is associated with the encoder-model lineage and its `##` convention. - **Unigram** is preferred where multilingual balance and subword regularisation matter, and where the probability model itself is useful. An interviewer asking this question wants to hear that you know the criterion differs — count, likelihood gain, likelihood loss — and that encoding differs correspondingly — merge replay, longest match, probabilistic search. Claiming they are "basically the same thing with different names" is the answer that fails.