Why does Japanese text cost 2-3x more LLM tokens than its English translation?

## Fertility, defined Fertility is the average number of tokens a tokenizer produces per unit of text — per word, per character, or per unit of meaning, depending on which comparison you want. It is the single number that explains why two texts saying the same thing bill differently. A tokenizer learns its vocabulary from a corpus; sequences that appeared often enough become single vocabulary entries, and everything else is assembled from smaller pieces. Whatever was rare in that corpus is expensive at inference time, forever. ## Why non-English text pays more Pretraining and tokenizer-training corpora are overwhelmingly English and, more broadly, Latin-script. So English function words, common stems and frequent suffixes each occupy a vocabulary slot, and a paragraph of English prose lands around 3.5–5 characters per token. Japanese gets much less of that treatment: fewer of its character sequences earned dedicated entries, so text splits into short pieces, often near one token per character. Worse, modern tokenizers are byte-level, meaning any character with no learned merge decomposes into its UTF-8 bytes — and a single Japanese character is three bytes, so an unlucky rare character can cost three tokens by itself. The comparison people actually feel is not per character but per *meaning*. Japanese writes densely: a sentence that takes eighty English characters may take thirty Japanese characters. Even so, the token count typically lands two to three times higher, because the per-character ratio moves against Japanese by more than the density moves for it. ## The concrete failure this produces Consider a customer-support product that priced itself per conversation using the ~4-characters-per-token rule and a sample of English transcripts. Ship the same product in Japan and each conversation carries two to three times the tokens for the same exchange. Gross margin on the Japanese tier collapses; nothing in the code changed and no alert fires, because the system is behaving exactly as built. The same arithmetic hits capacity: history-trimming tuned so that thirty English turns fit will hold roughly ten to fifteen Japanese ones, so the assistant starts forgetting earlier in the conversation for one set of users than another. ## Content type matters as much as language Fertility is not only a language property. Measure a mixed corpus and you get four different regimes. English prose sits near four characters per token. Ordinary source code sits near 2.5–3.5 — punctuation, operators and indentation runs are their own tokens, and `camelCase` or `snake_case` identifiers split at the case or underscore boundary. Minified JavaScript or TypeScript is worse than readable code, because mangled one- and two-character names share no learned merges. And a column of UUIDs, hex digests or base64 blobs is the floor: random character sequences have no statistical structure to compress, so they land near one to two characters per token. A retrieval system that stuffs raw identifiers into context is paying near the worst possible rate for text that carries little meaning. ## Why this is an equity issue, not just a billing quirk If price is per token, speakers of under-represented languages pay more for the same service, wait longer for the same answer, and get a smaller effective context. That asymmetry sits below the product, in the tokenizer, and no amount of prompt tuning removes it. The honest framing in an interview is that the gap is a known and measurable property of the vocabulary, that it has narrowed as vocabularies grew from tens of thousands to hundreds of thousands of entries and as multilingual data share increased, and that it has not disappeared. ## What to actually do Measure, do not assume. Take a representative sample per locale and per content type from your own traffic and compute characters per token for each; that table, not a blog-post constant, is what your cost model and your budgeting code should use. Price and forecast per locale. When choosing between models, include tokenizer fertility on your languages in the comparison — a model with a slightly higher per-token price can be cheaper end to end if it segments your text more efficiently, and the reverse trap is just as common. Where content is machine-generated, reduce fertility at the source: send identifiers by reference rather than pasting them, strip boilerplate, and avoid re-sending large blobs each turn. Finally, budget in tokens everywhere, so that the fertility difference is absorbed by the budget rather than discovered as a production incident.