Why does an English-trained embedding model degrade on Portuguese and Japanese text?

## The failure is quiet, which is the point Run a catalogue of Portuguese and Japanese product reviews through a strong English-trained embedding model and everything appears to work. Vectors come back at the right dimensionality. Similarity scores land in a familiar range. Search returns ten results per query, ranked. The only thing wrong is that the ranking is close to arbitrary, and no component of the system can tell you that. This is why language is a **first-order selection criterion**, decided before any leaderboard comparison — not something you discover in production. ## Mechanism one: training distribution A model's vector space is a learned map of the text it saw. Where it saw a lot, the map has fine structure: near-synonyms separate, domain senses separate, paraphrases collapse together. Where it saw little, the map is coarse — everything in that language crowds into an under-differentiated region, so distances between two Japanese reviews carry far less information than distances between two English ones. A useful mental check: the model cannot distinguish meanings it was never trained to distinguish. Absence of a language in training is not a mild handicap; it removes the very structure that similarity search reads. ## Mechanism two: the tokenizer Subword vocabularies are fitted to a corpus. Fit one on mostly-English text and it will contain whole common English words and useful English morphemes, but only fragments for other languages. - **Portuguese**, sharing the Latin script, fares better than you might fear at the character level but still fragments: accented forms, verb inflections and common function words split into pieces the model treats as unfamiliar units. - **Japanese** is worse on two counts. The script is outside the vocabulary's focus, so text may fall back to byte-level or very short pieces, and Japanese has no whitespace word boundaries, so the tokenizer's segmentation may not align with meaningful units at all. Two consequences follow. Quality drops because the model reasons over unfamiliar fragments. And the effective input shrinks: the same sentence costs several times more tokens, so more of it risks running past the model's length limit. ## Mechanism three: cross-lingual alignment This one is distinct and often missed. Even a model that handles each language *individually* will not necessarily place a sentence near its translation unless it was trained to. **Cross-lingual alignment** means exactly that: "the delivery was late" and its Portuguese equivalent occupy nearly the same point in one shared space. Models get that property by training on parallel or paired multilingual data with a contrastive objective. Without it, your mixed-language corpus is effectively several disjoint indexes wearing one coat: a Portuguese query retrieves Portuguese documents at best, never the English document that answers it. Decide up front which you need. "Each language searched within itself" is a weaker requirement than "any query retrieves any language", and only the second demands genuine alignment. ## What to do instead **Choose a model trained for your languages.** Multilingual encoders exist across the size range and are trained with explicit multilingual and often parallel data. Check the model's documented language list, not just its headline score. **Evaluate per language, not in aggregate.** A single averaged benchmark number hides exactly the failure you care about. Build a small labelled set — a few dozen query/relevant-document pairs per language, drawn from your own corpus — and measure retrieval quality in each. This is the only reliable evidence, and it is cheap. **Watch the tokenizer explicitly.** Compare tokens-per-character across your languages with the model's own tokenizer. A language costing three or four times more tokens than English is a warning about both quality and length budget. **Weigh one multilingual model against per-language models.** A multilingual model spends fixed capacity across many languages, so at equal size it is often slightly weaker on any single language than a dedicated one — a trade-off sometimes called the curse of multilinguality. Against that, one model means one index, one operational path, and cross-lingual retrieval for free. Per-language models mean better per-language quality but language detection at ingest and query time, several indexes, and no cross-lingual matching. Most teams take the multilingual model unless one language dominates the traffic. **Do not translate as a reflex.** Machine-translating everything into English before embedding is a real option and sometimes wins, but it adds a lossy step, a cost per document, and a dependency, and it does not help when the source text is short, colloquial or full of product-specific terms — exactly the character of user reviews. ## What an interviewer is testing That you know embedding quality is a property of training coverage rather than an intrinsic property of the model, that you can name both the tokenizer and the alignment mechanisms, and above all that you would catch this with a per-language evaluation instead of trusting that no errors means it works.