Your 80M stored 2048-dimension embeddings show only about 30 large singular values — what happened?

## Measuring it before naming it Take a large random sample of the stored vectors, say a few hundred thousand rows, and stack them into a matrix. **Mean-centre the columns** — skipping this is the single most common mistake, because an uncentred set with a large common offset shows one enormous leading singular value that is just the mean, not structure. Then take the singular values `s_1 >= s_2 >= ...`. Squared and normalised to sum to one, they give the share of variance in each orthogonal direction. A useful single number is the **effective rank**: normalise the squared singular values into a distribution `p_i`, compute its entropy, and report `exp(entropy)`. This is a smooth count of *how many directions actually participate*. Alternatively, report the number of directions needed to reach 99% of the variance — cruder, but easier to explain in a review. Either way, pin the sample size and the centring so the number is comparable between training runs, and track it as a training-time metric rather than discovering it in production. When effective rank is 30 out of a declared 2048, the representation has undergone **dimensional collapse**: not the degenerate case where every vector is identical (rank one, complete collapse), but the partial case where the vectors span a thin subspace of the space you are paying to store. ## Why the objective sets the rank The declared width is a ceiling. What fills it is the training signal. With supervised classification over `C` classes, the head is a `C x d` matrix and the features receive gradient **only** through it. The loss is satisfied as soon as the `C` class groups are linearly separable, and separating `C` groups needs at most `C - 1` directions of between-class structure. Within-class variation earns nothing and, in the late phase of training, actively shrinks — the published *neural collapse* phenomenon describes exactly this: class means converge toward a maximally-separated simplex configuration and within-class scatter goes toward zero. A 32-category label set producing about 30 usable directions is not a coincidence; it is the objective working as specified. Two other causes are worth checking before you blame the head. **Under-diverse data**: if the training set is narrow, the features never needed to spread. And **a collapse-prone unsupervised objective**: an objective that only pulls similar things together, with no repulsion, no negatives and no explicit variance or decorrelation term, has a trivial optimum where the encoder outputs a constant. Methods in that family carry an anti-collapse mechanism precisely because of this. ## Why it matters here and when it does not The key judgment — and the one interviewers are actually testing — is that **low effective rank is not bad in the abstract; it is bad relative to the task**. If the only downstream use were the same 32-way categorisation, 30 directions is exactly right and the collapse is efficient compression. It is bad here because the marketplace task is *instance-level*: detect that this listing photo is a re-post of that one, among 80M items in the same handful of categories. Instance identity is precisely the within-class variation the objective was told to discard. So the symptom in production is a similarity distribution where every pair of items in a category scores high, duplicates and non-duplicates overlap heavily, and no threshold separates them cleanly. You are also paying for 2048 stored coordinates per item to carry 30 directions of information. ## What to do about it **Do not widen the layer.** Going to 4096 doubles storage for the same 30 directions; width was never the binding constraint. **Change the training signal.** In rough order of cost: introduce finer supervision — attribute labels, a deeper taxonomy, or instance-level identity where you have it, which directly asks the features to keep within-category differences. Or train the encoder with a self-supervised objective that treats two augmented views of the same photo as a positive pair, which makes instance identity the training target and typically produces a much fuller spectrum, provided it includes negatives or an explicit variance/decorrelation regulariser. **Consider an earlier layer as a stopgap.** Layers further from the head are less collapsed by the projection and often retain more variance, at the cost of less semantic organisation. **Then, and only then, revisit width.** Once effective rank is measured, shrinking the stored vector toward it is nearly free in quality and a large win in storage — but understand that this is a cost optimisation, not a quality fix. Reducing 2048 collapsed dimensions to 64 does not add information; it merely stops paying for emptiness. ## The honest limit of the diagnostic A healthy spectrum is necessary, not sufficient. High effective rank only says variance is spread across many directions; those directions may encode lighting, compression artefacts, watermarks or background rather than anything about the object. Always pair the spectrum with a task metric — duplicate precision and recall at your operating threshold — and treat effective rank as the cheap early warning, not the verdict.