In Sentence Transformers, which loss fits unlabelled (question, answer) pairs, pairs with a similarity score, and (anchor, positive, negative) triplets, and why?
answer
- the loss defines similar
- in-batch negatives need no labels
- ranking pairs by their score
- a Euclidean margin of 5
- the loss overview table
basics
~10 sUnlabelled pairs: MultipleNegativesRankingLoss, which uses the rest of the batch as negatives. Scored pairs: CoSENTLoss or CosineSimilarityLoss. Triplets: MultipleNegativesRankingLoss again, with the negative added, or TripletLoss with a tuned margin.
solid answer
~40 sThe loss is where you say what "similar" means, so I pick it from the shape of the labels. For `(question, answer)` pairs with no labels, `MultipleNegativesRankingLoss` is the documented default: each question must score its own answer above every other answer in the batch, which is exactly a retrieval objective. For pairs with a float score, `CoSENTLoss` (or `AnglELoss`) learns to order pairs by their score and is recommended over the older `CosineSimilarityLoss`, which regresses the cosine onto the label with mean squared error. For triplets, `MultipleNegativesRankingLoss` still fits and adds the explicit negative to the in-batch ones, while `TripletLoss` only pushes the negative a margin further away than the positive, by default 5 in Euclidean distance. For 0/1 pair labels there are `ContrastiveLoss` and `OnlineContrastiveLoss`.
go deeper
Recall the main mapping: unlabelled pairs take MultipleNegativesRankingLoss, scored pairs take CoSENTLoss or CosineSimilarityLoss, triplets can use either ranking or TripletLoss.
Explain what each loss optimises: in-batch ranking, score ordering, or a distance margin, and why CoSENTLoss is preferred to MSE on cosine for graded labels.
Justify choosing a ranking loss for retrieval data even when triplets exist, and know the TripletLoss defaults well enough to spot a margin that can never be met.
Frame the choice as what the product treats as similar, and decide whether to reshape labels, mine negatives or collect new annotations before settling for a loss that only fits the data you happen to have.
## Why the loss is the real design decision Fine-tuning an embedding model with `SentenceTransformerTrainer` changes one thing about the model: which inputs end up close together. The **loss function** is where that intent is written down. The same base model trained on the same texts becomes a retriever, a paraphrase scorer or a clustering model depending on the loss, and every loss expects a specific **data shape**: a number of input columns and, sometimes, a label column. The library's loss overview is organised exactly that way, inputs and labels first, loss second. ## The common shapes and their losses | Inputs | Label | Losses to reach for | What they optimise | |---|---|---|---| | `(anchor, positive)` pairs | none | `MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss` | each anchor ranks its own positive above every other candidate in the batch | | `(anchor, positive, negative)` triplets | none | `MultipleNegativesRankingLoss`, `TripletLoss` | ranking with an extra hard negative, or a margin between two distances | | `(anchor, positive, negative_1, ..., negative_n)` | none | `MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss` | ranking with several hard negatives per anchor | | `(input_A, input_B)` pairs | float score | `CoSENTLoss`, `AnglELoss`, `CosineSimilarityLoss` | similarity that tracks a graded score | | `(anchor, positive/negative)` pairs | 1 or 0 | `ContrastiveLoss`, `OnlineContrastiveLoss` | pull matches together, push non-matches apart | | `(input_A, input_B)` pairs | class | `SoftmaxLoss` | a classifier over the pair's embeddings | ## Unlabelled pairs: MultipleNegativesRankingLoss This is the starred default for retrieval data such as question-answer pairs: - For every anchor it computes the similarity (cosine by default, multiplied by `scale`, default 20) to **every positive in the batch**, and to any extra negative columns. - It applies a cross-entropy whose correct class is the anchor's own positive. The other rows' positives act as **in-batch negatives**, so no labels and no mined negatives are needed. - Because the negatives come from the batch, the batch size changes what it learns, which is why `CachedMultipleNegativesRankingLoss` exists to reach very large batches. - The library also documents it under its literature names: InfoNCE, or the in-batch negatives loss. ## Scored pairs: CoSENTLoss over CosineSimilarityLoss `CosineSimilarityLoss` computes the cosine of the two embeddings and minimises the mean squared error against the label, so it expects labels in `[0, 1]`. `CoSENTLoss` instead looks at all pairs of pairs in the batch and penalises any case where a pair with a lower label scores higher than a pair with a higher label. It optimises the **ordering**, not the absolute value, and the library's own docstrings report it as a stronger training signal and a drop-in replacement. `AnglELoss` is `CoSENTLoss` with an angle-based similarity. ## Triplets: two very different objectives 1. **`MultipleNegativesRankingLoss` on triplets** treats the third column as an extra hard negative for that anchor, on top of all the in-batch negatives. It is still a ranking objective and is the recommended choice. 2. **`TripletLoss`** computes `max(d(anchor, positive) - d(anchor, negative) + margin, 0)`. Its defaults are `TripletDistanceMetric.EUCLIDEAN` and `triplet_margin=5`, and the docstring warns that the margin must be tuned. Switching to `TripletDistanceMetric.COSINE` without lowering the margin gives a margin no cosine distance can satisfy. ## Matching loss and data in practice - Reshape before you compromise on the loss: class-labelled pairs can be sampled into triplets, and pairs can be enriched with mined hard negatives. - Remember the dataset contract: input columns are read by position and the label must be named `label`, `labels`, `score` or `scores`. - Losses such as `MatryoshkaLoss` are **modifiers** that wrap one of the losses above rather than a choice of their own. - Pass the loss explicitly; the trainer's fallback, `CoSENTLoss`, only fits scored pairs. ## Checking that the choice worked - Loss values are not comparable across losses: a `CoSENTLoss` of 2.1 and a `MultipleNegativesRankingLoss` of 0.4 say nothing about which model retrieves better. - Attach a task evaluator that matches the use case, such as `InformationRetrievalEvaluator` for retrieval or `EmbeddingSimilarityEvaluator` for scored pairs, and compare runs on its metric. - If you have several data shapes, you do not have to pick one loss: a dict of datasets with a matching dict of losses trains each shape with its own objective in the same run. The interview point is the mapping from label shape to objective: pairs without labels mean a ranking loss, graded scores mean an ordering loss, binary labels mean a contrastive loss, and a triplet dataset does not oblige you to use `TripletLoss`.
- Your duplicate-ticket labels are 1 for duplicate and 0 for not. Which Sentence Transformers losses fit, and is there another route?`ContrastiveLoss` or `OnlineContrastiveLoss` take `(anchor, other)` pairs with a 1/0 label directly. Alternatively, keep only the positive pairs and train `MultipleNegativesRankingLoss`, optionally using the 0-labelled partners as an explicit negative column, which turns the task into ranking.
- What does the scale argument of MultipleNegativesRankingLoss do, and when would you change it?The similarities are multiplied by `scale` before the softmax; it is the inverse of a temperature, so the default 20 equals a temperature of 0.05. A higher scale sharpens the focus on the positive. If you switch `similarity_fct` to a dot product, the docstring advises a scale of 1.
- Can MultipleNegativesRankingLoss use more than one hard negative per anchor?Yes. It accepts `(anchor, positive, negative_1, ..., negative_n)` rows, adding every negative column to the candidates for that anchor. The number of columns must be fixed across the dataset; a column holding a variable-length list of negatives is not supported for `SentenceTransformer` training.
- CosineSimilarityLoss is the right loss for unlabelled question-answer pairs.
- MultipleNegativesRankingLoss needs an explicit negative in every row.
- TripletLoss defaults to cosine distance with a small margin like 0.1.
- Any loss works as long as the dataset has two text columns.
- A triplet dataset means you have to train with TripletLoss.