What do you lose when you truncate a Matryoshka embedding from 3072 to 256 dimensions?

## What Matryoshka training actually does Matryoshka Representation Learning (MRL, introduced in 2022) changes the training objective, not the architecture. A normal encoder is trained with a loss computed on the full output vector; MRL computes the loss at several nested prefix lengths at once — say 64, 128, 256, 512, 1024, and the full width — and sums them. The gradient therefore pushes the model to make the *first* 64 values a self-sufficient embedding, the first 128 a slightly better one, and so on, like nested dolls. Information is deliberately ordered by importance along the dimension axis. The consequence is that one stored 3072-dimension vector is really a family of embeddings. Slicing off a prefix is free at query time — no re-encoding, no separate model, no second index build from scratch. ## Why you cannot do this to an ordinary embedding In a conventionally trained encoder, no dimension is privileged. Meaning is distributed across the whole vector, and the ordering of the output units is an artefact of initialisation. Taking the first 256 values of a 3072-dimension conventional embedding discards an arbitrary 92% of the representation and typically collapses retrieval quality. This is the single most important distinction in the topic: **truncation is a property the model was trained to support, not an operation you may apply to any vector.** Reducing a conventional embedding requires a fitted transform such as PCA, which is a different technique with different validation needs. ## What is actually lost Think of the prefix as a lower-resolution image of the same scene. At 256 dimensions: - **Coarse separation survives.** Documents from clearly different topics stay far apart. Broad queries barely move. - **Fine discrimination degrades.** Distinguishing two documents that differ in a qualifier, a negation, an entity, or a numeric detail depends on subtler directions that live in the discarded tail. - **The loss is concentrated, not uniform.** Aggregate recall@10 may drop only a point or two while the hard slice of your query set drops far more. Reporting only the mean hides this. - **Score distributions shift.** Absolute similarity values at 256 dimensions are not comparable to those at 3072, so any hard-coded cutoff you tuned on full-width vectors must be re-tuned. ## The mandatory re-normalisation step Embedding APIs usually return unit-length vectors. A prefix of a unit vector is not unit length — its norm is the square root of the retained energy. If your pipeline assumes normalised vectors so that a dot product equals cosine similarity, you must L2-normalise after slicing. Forgetting this yields scores biased by how much magnitude each document happened to keep in its leading dimensions, which quietly corrupts ranking. Interviewers ask about this because it is the step people skip. ## Worked example: on-device recipe search Suppose you ship a recipe app that must search 200,000 recipes entirely on the phone, with a hard budget of a few hundred megabytes of RAM. At 3072 dimensions and float32, the vectors alone are 200,000 x 3072 x 4 = about 2.5 GB — impossible. Truncated to 256 dimensions the same corpus is about 205 MB, which fits with room for the index. The method is not to guess. Embed the corpus once at full width, keep it server-side as ground truth, then evaluate prefixes at 128, 256, 512, and 1024 against a labeled query set drawn from real app searches. You will usually see recall@10 nearly flat down to some point and then fall off a cliff. Pick the smallest prefix on the flat part, with a margin. If 256 sits just past the cliff and 384 sits comfortably before it, ship 384 — the memory difference is small and the quality difference is not. ## Two-pass designs recover most of the loss When you control both stages, you do not have to accept the truncated ranking as final. Store both widths, shortlist with the 256-dimension vectors — cheap, small, cache-friendly — then rescore only the top 200 candidates with the full 3072-dimension vectors. Recall is dominated by the shortlist stage, precision at the top of the list by the rescore stage, and you pay full-width comparison cost for a few hundred documents rather than the whole corpus. The trade is a second lookup and holding both representations, which usually beats holding only the full-width one. ## Trained-in versus bolted-on The wider framing an interviewer is often fishing for: MRL is dimensionality reduction moved *into* training, so the model itself decides what to sacrifice first, whereas PCA and similar post-hoc projections are fitted afterwards on a sample of vectors and can only exploit the structure that happens to be there. Trained-in nesting generally degrades more gracefully; post-hoc reduction is what you reach for when the model you are stuck with was not trained for nesting.