Why do LLMs tokenize text into subwords instead of whole words or characters?

## The problem a tokenizer solves A transformer does not consume text. It consumes a sequence of integers, each of which indexes a row in an embedding matrix. That matrix has a fixed number of rows, chosen before pretraining starts and unchangeable afterwards without surgery on the model. So tokenization is the question: how do you map an unbounded space of possible strings onto a bounded set of ids, reversibly and cheaply? ## Word-level tokenization and the unknown-word wall The intuitive answer is a dictionary of words. Split on whitespace and punctuation, assign each distinct word an id, keep the most frequent N. This fails in three ways at once. First, coverage: natural text has a long tail. Names, misspellings, hashtags, chemical formulae, filenames and camelCase identifiers appear constantly and were not in the dictionary. Everything outside the list becomes a single unknown token, so `Zyprexa` and `qwertyuiop` become literally the same input, and the model can neither read nor generate them. Second, morphology: `run`, `runs`, `running`, `runner` occupy four unrelated rows with no shared structure, so anything learned about one is not shared with the others. Languages that build words by stacking suffixes make this far worse. Third, size: covering enough of the tail requires hundreds of thousands of entries, and each entry costs a full embedding row. ## Character-level tokenization and the length wall The opposite extreme is one token per character. The vocabulary is tiny, nothing is ever unknown, and morphology is fully exposed. But the sequence gets much longer — English averages roughly four to five characters per common subword token, so the same document becomes several times as many positions. Attention cost grows with sequence length, the usable context shrinks by the same factor, and the model has to spend capacity relearning that `t`,`h`,`e` is a word before it can do anything semantic. Character models exist and work, but they are not the economical choice at frontier scale. ## The subword compromise Subword tokenization takes the useful half of each. The vocabulary is built from a base alphabet — every character in the corpus, or in byte-level variants all 256 possible byte values — plus a set of learned multi-character pieces. Learning is data-driven: the pieces that earn a vocabulary slot are the ones that recur often enough in the training corpus to pay for themselves. The result has three properties interviewers expect you to name: - **Frequent words stay whole.** `the`, `because`, `government` are single tokens, so common text is compact. - **Rare words decompose into reusable pieces.** An unseen word is spelled from pieces the model has seen elsewhere, so some of its structure survives — a prefix like `anti`, a suffix like `ation`. - **Nothing is truly out of vocabulary,** because the base alphabet is always available as a fallback. The named algorithms differ in how they choose which pieces get a slot — byte-pair encoding merges the most frequent adjacent pair repeatedly, WordPiece scores candidate merges by how much they improve corpus likelihood, and Unigram starts from a large candidate set and prunes it — but all three land in the same design space. ## What follows from this Two consequences are worth internalising early. One: the tokenizer is a separate artefact, trained on its own corpus before the model, and frozen for the model's lifetime. You cannot swap it after pretraining, because the ids are baked into the embedding matrix. Two: token boundaries are an accident of corpus statistics, not of meaning. A word that was frequent in the tokenizer's training data is one token; a semantically identical word that was rare is five. That asymmetry propagates into everything downstream — how much text fits in context, how well the model handles a given language or a given code style, and why it sometimes appears blind to the individual letters inside a word it has never had to spell out. ## Getting the framing right in an interview The strongest short answer names both failure modes, not one. Candidates who only say "so we can handle unknown words" are half right and miss why we do not simply use characters; candidates who only say "to keep sequences short" miss the coverage argument. Say both, then name the mechanism — a learned vocabulary of frequent pieces over a guaranteed base alphabet.