How does SigLIP's sigmoid loss differ from CLIP's softmax contrastive objective?

## What both objectives are trying to do Contrastive image-text pretraining takes a very large corpus of (image, caption) pairs, runs the image through an image encoder and the caption through a text encoder, and trains both so that matching pairs have high similarity and mismatched pairs have low similarity. The result is an image encoder whose features are *semantically* organized — an encoder that has learned what things are, because it had to distinguish captions describing them. Every mainstream VLM uses such an encoder as its vision backbone. ## CLIP: softmax over the batch CLIP computes a similarity matrix between all N images and all N captions in a batch, then applies a cross-entropy loss in both directions: for each image, the correct caption should win a softmax over the N captions; for each caption, the correct image should win over the N images. There is a learned temperature scaling the logits. The consequence sits in the denominator. Because the softmax normalizes over the whole batch, the loss for one pair depends on *which other pairs happen to be in the batch*. Two things follow: - **Batch size is a quality knob.** More in-batch negatives means a harder discrimination task and a better signal. CLIP-style training pushed batch sizes to tens of thousands. - **Distributed training needs synchronization.** With embeddings split across many accelerators, computing the full similarity matrix requires all-gathering embeddings, and memory for the NxN matrix grows quadratically. This is engineering overhead purely to satisfy the objective's shape. ## SigLIP: pairwise sigmoid SigLIP keeps the encoders and the data and changes only the loss. Each (image, text) pair is treated as an independent binary classification: is this a real pair or not? A sigmoid is applied to the scaled similarity, with a learned temperature and a learned bias term that compensates for the heavy imbalance between the few positives and the many negatives. Because there is no softmax denominator, no normalization crosses pair boundaries. That yields: - **No global batch dependence.** The loss for a pair is well defined on its own, so training behaves sensibly at modest batch sizes where CLIP-style training degrades. - **Simpler scaling.** Negative pairs can be handled in a chunked, device-local fashion instead of materializing a global similarity matrix, so memory and communication drop. - **Better encoders in practice.** SigLIP models match or beat CLIP counterparts at equal compute, which is why they became the standard backbone. ## SigLIP 2 and what it added SigLIP 2 kept the sigmoid loss as the core and layered on additional training signals — captioning-style decoder pretraining and self-distillation / masked-prediction objectives — aimed at improving *dense* features, meaning the quality of the per-patch representations rather than just the pooled global vector. That matters directly for VLMs, because a VLM consumes the per-patch outputs, not the pooled one. It also broadened the text side to be multilingual and shipped variants that accept native aspect ratios and variable sequence lengths rather than forcing a fixed square. ## What a contrastively pretrained encoder does and does not encode This is the part interviewers actually probe, because it explains VLM failure modes. **Encoded well:** object and scene identity, style, coarse attributes, broad text-image semantics. The training signal is "which caption goes with this image", and captions carry exactly that. **Encoded poorly:** - **Compositional relations.** "The red cube left of the blue sphere" and "the blue sphere left of the red cube" share almost all their words. A bag-of-concepts representation matches both about equally, and the objective rarely punishes the confusion, so contrastive encoders are notoriously weak at binding attributes to objects and at spatial relations. - **Exact counts.** Captions say "some birds" far more often than "seven birds", so counting is barely supervised. - **Fine text and small structure.** At typical pretraining resolutions, small glyphs occupy less than a patch, so the representation has nowhere to put them. - **Negation.** Very little training text asserts what is absent. These gaps are inherited by the VLM built on top. The decoder can only reason over what the encoder handed it, so "the model can't count the items" or "it confuses which object is on the left" is often an encoder-representation limit, not a prompting problem. Mitigations live elsewhere in the stack: higher input resolution and tiling for the detail gaps, and reasoning-time cropping and re-inspection for counting and relations. ## The compact answer CLIP: softmax over in-batch pairs, global normalization, big synchronized batches. SigLIP: independent sigmoid per pair, no global normalization, cheaper and better-behaved. Both produce encoders that are strong on "what is in this image" and weak on "how are these things arranged and how many are there".