An embedding lookup meets an id it never saw in training -- what happens at serving time?

## The vocabulary is frozen at training time An embedding table has a fixed number of rows, chosen before training from the ids observed in the training data, together with an **id-to-index map** that turns a raw id (a postal code, a merchant string, a product SKU) into a row number. Both the table and the map are part of the model artifact. Nothing about the layer can invent a row for an id it has never indexed, because a new row would be a new parameter with no trained value. So when a postal code that never appeared in training arrives at serving time, one of three things happens, and which one is a design decision you made -- knowingly or not: 1. **The index is out of range.** The lookup fails, and the request errors. Loud, and honestly the least dangerous outcome, because you find out immediately. 2. **The map silently produces something wrong.** A map that returns a default of `0` without `0` being a reserved row will hand back whichever id happens to own row 0. Silent, and the model scores the request confidently with an unrelated id's vector. 3. **A reserved unknown row answers.** The vocabulary keeps one index -- conventionally `0` -- for "any id I do not know", and every unseen id maps there. Only the third is a design. The interviewer is usually checking whether you know that, and whether you know the catch in it. ## The catch: a reserved row that was never trained is not a fallback Reserving row `0` costs one line of vocabulary code and buys nothing on its own. If no training example ever routes to that row, it receives **zero gradient for the entire run** and finishes at its random initialization -- a vector from a distribution the downstream layers were never trained to consume. The model does not degrade gracefully to some average prediction; it evaluates a network on an input it has never seen, and the output is arbitrary. Two standard ways to give that row a meaning, usually used together: - **A minimum-frequency cut.** Choose a threshold -- ids seen fewer than, say, five times in training -- and map all of them to the unknown row during training. The row then learns something like "a rare id I have no information about", which is exactly the situation at serving. - **Random id dropout.** Route a small random fraction of *in-vocabulary* ids to the unknown row on each pass. This is the same trick as dropout applied to ids: it trains the rest of the network to still produce a sensible score when the id contributes nothing, so the model leans on the other features rather than collapsing. The second point generalises: cold start is a **feature** problem as much as an embedding problem. A model that can only score a request through its id embedding has nothing to say about a new id. A model that also takes side features -- region, device, category, account age -- degrades to "a typical entity with these attributes", which is a usable prediction. ## The rare-id problem is the same problem, one notch weaker An id seen three times in training does have a row, and that row is nearly useless: three gradient updates leave it close to its initialization, and whatever it did learn is fitted to three examples. If the optimizer applies weight decay densely, the row is also being pulled toward zero on every step between its rare appearances. Yet the model treats that row with the same confidence as a row trained on a million examples. This is why the minimum-frequency cut is not only about unseen ids. Folding the whole thin tail into one shared row (or a handful of frequency-bucketed rows) trades per-id resolution you never had for a row that is actually estimated. In a recommender where 90% of user ids appear fewer than five times, per-id rows for that 90% are mostly parameters memorising noise -- while contributing the bulk of the table's memory. ## Hashing: the alternative that has no unknown case Instead of a learned id-to-index map, hash the raw id into a fixed number of buckets, `index = hash(id) mod B`. Any id, seen or unseen, gets an index, so the unknown case disappears by construction and the table size is decoupled from cardinality. The price is **collisions**: two distinct ids landing on one row share a single vector and become indistinguishable to the model. With `B` well above the number of *frequent* ids, most collisions pair a frequent id with a rare one, and the damage is concentrated where you had little signal anyway. Splitting an id across two smaller tables with different bucketings and combining the two rows reduces the chance that two ids collide in both, at the cost of a slightly more complex lookup. The trade is: a learned vocabulary gives clean per-id rows plus an explicit unknown path and a map you must version and ship; hashing gives no unknown path and no map, at the cost of some ids sharing capacity. ## The production bug this question is really probing A vocabulary rebuilt at serving time -- from live data, from a fresher dump, from a job that runs on its own schedule -- shifts indices. The lookup never errors, every id resolves, and every id resolves to a row trained for a *different* id. Metrics drop, nothing throws, and the cause is invisible in the model code. Ship the id-to-index map inside the model artifact, version it with the weights, and check its fingerprint at load. Then instrument the **unknown rate** at serving: a rate that is stable tells you the vocabulary still matches the traffic, and a rate that climbs is a drift alarm that fires before the metric does.