An English-trained tokenizer shatters Finnish words into many pieces — why, and what fixes it?

## The mechanism, not the language It is tempting to say agglutinative languages are "hard to tokenize". They are not. A merge table is a compression code fitted to a corpus, and a code fitted to English is simply a bad code for Finnish — the same way a Huffman table built from English letter frequencies compresses Finnish poorly. Every merge that exists is one that paid for itself in the training text. If Finnish, Turkish, Hungarian or Tamil made up a fraction of a percent of that text, their recurring morphemes never accumulated enough count to beat English candidates, and the vocabulary reflects that. Agglutination amplifies the effect. A Finnish or Turkish word may carry case, number, possession and several derivational suffixes in one orthographic word. In English those would be separate whitespace-delimited words, each with its own chance to earn a token. Stacked into one word, each *combination* is individually rare, so none of them earns a slot even if each morpheme is common. ## What the symptom looks like The observable is **fertility**: tokens produced per word. A well-served language sits near one to one-and-a-half tokens per word; a badly served one can run to five or eight. Three things degrade together: - **Context.** The same document occupies several times more positions, so less of it fits in the window and long-document tasks fail earlier. - **Compute and latency.** More tokens in and more tokens out, for identical content. - **Quality.** The model must compose meaning from unstable fragments whose boundaries shift with inflection, so the same stem is represented differently in different forms. Downstream, the language's tokens are also individually rarer during pretraining, so they receive fewer gradient updates. This is why the same model can look competent in one language and clumsy in another even when both were present in the pretraining data. ## Fixing it before pretraining At tokenizer-training time the levers are direct and cheap: 1. **Rebalance the tokenizer corpus.** The tokenizer is fit on a sample, and that sample does not have to mirror the pretraining mix. Upsampling target languages buys them merges. A common approach is sampling languages with a temperature that flattens the distribution rather than reproducing raw web proportions. 2. **Size the vocabulary for the language count.** Serving many scripts well from 32K slots is arithmetically hard; larger vocabularies exist substantially to buy coverage across languages and code. This trades against embedding and output-layer cost. 3. **Keep byte-level fallback.** It guarantees nothing is lost while you are still under-serving a language, converting a correctness problem into a cost problem. 4. **Measure per language and gate on it.** Tokens-per-word or tokens-per-character by language, computed on held-out text, is a cheap acceptance test and the thing nobody runs until a user complains. ## Fixing it after pretraining Once the model is trained, the tokenizer is effectively frozen: ids index embedding rows, so a different tokenizer means the ids mean different things and the model is destroyed. Dropping in a replacement is not an option, and any candidate who proposes it has revealed they do not know how ids bind to weights. What is actually done is **vocabulary extension with continued pretraining**: add new tokens for the target language's frequent sequences, grow the embedding and output matrices, initialise the new rows sensibly — a common choice is the mean of the embeddings of the pieces the new token replaces, which starts it in a plausible region rather than at random — and then continue pretraining on a mixture heavy in the target language until the new rows are trained and the old behaviour has not regressed. This is a real training run with real cost and real regression risk, which is exactly why the allocation decision deserves attention before pretraining rather than after. ## The judgement an interviewer is testing Three things separate a strong answer. First, correctly locating the cause in corpus statistics and vocabulary allocation rather than in the language. Second, knowing that the consequences are simultaneously about cost, effective context and quality, not just cost. Third, knowing that the remedy is cheap before pretraining and expensive after, and being able to describe the extension-plus-continued-pretraining path without pretending it is free.