Contrastive Self-Supervision

## The setup Contrastive self-supervision has no labels. It manufactures a supervision signal from the data itself: take one input, pass it through two random augmentations, and declare those two **views** a *positive pair*. Every other item in the comparison pool is a *negative*. An encoder maps each view to an embedding, the embeddings are L2-normalised, and similarity is cosine — the dot product of two unit vectors, bounded in [-1, 1]. ## The loss Write `s_p` for the anchor's similarity to its positive and `s_1 ... s_K` for its similarities to K negatives. InfoNCE (also called the NT-Xent loss in this setting) is ``` loss = -log( exp(s_p / T) / ( exp(s_p / T) + sum_j exp(s_j / T) ) ) ``` Read it as cross-entropy over a (K+1)-way softmax where the correct class is 'my other view'. Two things follow immediately. First, **the positive term is in the denominator too**. The loss is minimised not by making `s_p` large in absolute terms but by making it large *relative to* the negatives. An encoder that maps everything to one vector gets `s_p = s_j = 1` for all `j` and pays `log(K+1)` — the worst possible score. The negatives are exactly what makes the trivial constant solution unattractive. Second, the objective decomposes into two forces. The numerator pulls the two views of the same input together — **alignment**, an invariance to whatever the augmentations changed. The denominator pushes everything else apart — **uniformity**, a pressure to spread embeddings over the sphere so the encoder does not waste capacity. Good representations need both; either alone degenerates. ## What the temperature actually does `T` divides every similarity before the exponential. Because softmax is scale-sensitive, this is a hardness-weighting knob, not a learning-rate knob. The gradient of the loss with respect to negative `j`'s similarity is proportional to that negative's softmax probability, `p_j = exp(s_j/T) / sum_all exp(s/T)`. With `T` small, a negative that is only slightly closer than the others takes almost all of the probability mass, so almost all of the repulsive gradient lands on it. With `T` large, `p_j` flattens toward uniform and every negative contributes about equally. Concretely: a positive at cosine 0.8 against negatives at 0.7, 0.2 and -0.1. At `T = 0.5` the hardest negative (0.7) carries about 64% of the negative mass. At `T = 0.07` it carries 99.9% — the other two negatives have effectively been deleted from the loss. That has consequences in both directions. Low `T` gives sharp, well-separated clusters quickly, but it makes training sensitive to a handful of examples per step, amplifies label noise you cannot see, and maximally punishes false negatives. High `T` trains gently and tolerantly, but the embedding space stays diffuse and downstream probes underperform. Sweeping `T` from 0.07 to 0.5 while pretraining a wrist-worn accelerometer encoder for human-activity data is a genuinely different experiment from sweeping a learning rate: the low end sharpens the boundary between 'walking' and 'stair-climbing' segments that look nearly identical in raw acceleration, while the high end refuses to commit and leaves them overlapping. There is no domain-independent best value; it interacts with the augmentation strength and with how many negatives you have. ## False negatives InfoNCE assumes every non-positive is a true negative. It is not. Imagine pretraining on a 40-item product catalogue with many photos per item: two augmented crops taken from two *different* photos of the same item are semantically identical, and InfoNCE actively pushes them apart. The rate of such collisions rises with the number of negatives and with how few distinct concepts your corpus contains, and their damage rises as `T` falls, because a false negative is by construction a *hard* negative and low temperature hands it the whole gradient. Mitigations: pretrain on data diverse enough that collisions are rare, raise `T`, or promote confident nearest neighbours to additional positives. ## Practical notes - Normalise before computing similarity. Without unit vectors, `T` is competing with the embeddings' free-floating magnitude and stops meaning anything. - The loss is usually symmetrised: each view of the pair takes a turn as the anchor. - Contrastive training typically applies the loss to the output of a small projection network on top of the encoder, and then discards that projection, transferring the encoder's representation instead — the projection layer absorbs the invariances the loss demands and the layer beneath it retains information the loss would have thrown away. - The loss value is not comparable across different `T` or different negative counts. Compare runs by a downstream probe, never by the raw loss.