skip to content

Fine-tuning & Loss Functions

Adapting an embedding model with SentenceTransformerTrainer, where the loss encodes what similar means for your labels. Interviewers ask which loss fits pairs, triplets or scores, and why it matters.

on this pageshow

questions

6

In Sentence Transformers, which loss fits unlabelled (question, answer) pairs, pairs with a similarity score, and (anchor, positive, negative) triplets, and why?

level: middleimportance: must knowfreq 28%

answer

  1. the loss defines similar
  2. in-batch negatives need no labels
  3. ranking pairs by their score
  4. a Euclidean margin of 5
  5. the loss overview table

basics

~10 s

Unlabelled pairs: MultipleNegativesRankingLoss, which uses the rest of the batch as negatives. Scored pairs: CoSENTLoss or CosineSimilarityLoss. Triplets: MultipleNegativesRankingLoss again, with the negative added, or TripletLoss with a tuned margin.

solid answer

~40 s

The loss is where you say what "similar" means, so I pick it from the shape of the labels. For `(question, answer)` pairs with no labels, `MultipleNegativesRankingLoss` is the documented default: each question must score its own answer above every other answer in the batch, which is exactly a retrieval objective. For pairs with a float score, `CoSENTLoss` (or `AnglELoss`) learns to order pairs by their score and is recommended over the older `CosineSimilarityLoss`, which regresses the cosine onto the label with mean squared error. For triplets, `MultipleNegativesRankingLoss` still fits and adds the explicit negative to the in-batch ones, while `TripletLoss` only pushes the negative a margin further away than the positive, by default 5 in Euclidean distance. For 0/1 pair labels there are `ContrastiveLoss` and `OnlineContrastiveLoss`.

go deeper

for a junior

Recall the main mapping: unlabelled pairs take MultipleNegativesRankingLoss, scored pairs take CoSENTLoss or CosineSimilarityLoss, triplets can use either ranking or TripletLoss.

for a middle

Explain what each loss optimises: in-batch ranking, score ordering, or a distance margin, and why CoSENTLoss is preferred to MSE on cosine for graded labels.

for a senior

Justify choosing a ranking loss for retrieval data even when triplets exist, and know the TripletLoss defaults well enough to spot a margin that can never be met.

for a principal

Frame the choice as what the product treats as similar, and decide whether to reshape labels, mine negatives or collect new annotations before settling for a loss that only fits the data you happen to have.

## Why the loss is the real design decision Fine-tuning an embedding model with `SentenceTransformerTrainer` changes one thing about the model: which inputs end up close together. The **loss function** is where that intent is written down. The same base model trained on the same texts becomes a retriever, a paraphrase scorer or a clustering model depending on the loss, and every loss expects a specific **data shape**: a number of input columns and, sometimes, a label column. The library's loss overview is organised exactly that way, inputs and labels first, loss second. ## The common shapes and their losses | Inputs | Label | Losses to reach for | What they optimise | |---|---|---|---| | `(anchor, positive)` pairs | none | `MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss` | each anchor ranks its own positive above every other candidate in the batch | | `(anchor, positive, negative)` triplets | none | `MultipleNegativesRankingLoss`, `TripletLoss` | ranking with an extra hard negative, or a margin between two distances | | `(anchor, positive, negative_1, ..., negative_n)` | none | `MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss` | ranking with several hard negatives per anchor | | `(input_A, input_B)` pairs | float score | `CoSENTLoss`, `AnglELoss`, `CosineSimilarityLoss` | similarity that tracks a graded score | | `(anchor, positive/negative)` pairs | 1 or 0 | `ContrastiveLoss`, `OnlineContrastiveLoss` | pull matches together, push non-matches apart | | `(input_A, input_B)` pairs | class | `SoftmaxLoss` | a classifier over the pair's embeddings | ## Unlabelled pairs: MultipleNegativesRankingLoss This is the starred default for retrieval data such as question-answer pairs: - For every anchor it computes the similarity (cosine by default, multiplied by `scale`, default 20) to **every positive in the batch**, and to any extra negative columns. - It applies a cross-entropy whose correct class is the anchor's own positive. The other rows' positives act as **in-batch negatives**, so no labels and no mined negatives are needed. - Because the negatives come from the batch, the batch size changes what it learns, which is why `CachedMultipleNegativesRankingLoss` exists to reach very large batches. - The library also documents it under its literature names: InfoNCE, or the in-batch negatives loss. ## Scored pairs: CoSENTLoss over CosineSimilarityLoss `CosineSimilarityLoss` computes the cosine of the two embeddings and minimises the mean squared error against the label, so it expects labels in `[0, 1]`. `CoSENTLoss` instead looks at all pairs of pairs in the batch and penalises any case where a pair with a lower label scores higher than a pair with a higher label. It optimises the **ordering**, not the absolute value, and the library's own docstrings report it as a stronger training signal and a drop-in replacement. `AnglELoss` is `CoSENTLoss` with an angle-based similarity. ## Triplets: two very different objectives 1. **`MultipleNegativesRankingLoss` on triplets** treats the third column as an extra hard negative for that anchor, on top of all the in-batch negatives. It is still a ranking objective and is the recommended choice. 2. **`TripletLoss`** computes `max(d(anchor, positive) - d(anchor, negative) + margin, 0)`. Its defaults are `TripletDistanceMetric.EUCLIDEAN` and `triplet_margin=5`, and the docstring warns that the margin must be tuned. Switching to `TripletDistanceMetric.COSINE` without lowering the margin gives a margin no cosine distance can satisfy. ## Matching loss and data in practice - Reshape before you compromise on the loss: class-labelled pairs can be sampled into triplets, and pairs can be enriched with mined hard negatives. - Remember the dataset contract: input columns are read by position and the label must be named `label`, `labels`, `score` or `scores`. - Losses such as `MatryoshkaLoss` are **modifiers** that wrap one of the losses above rather than a choice of their own. - Pass the loss explicitly; the trainer's fallback, `CoSENTLoss`, only fits scored pairs. ## Checking that the choice worked - Loss values are not comparable across losses: a `CoSENTLoss` of 2.1 and a `MultipleNegativesRankingLoss` of 0.4 say nothing about which model retrieves better. - Attach a task evaluator that matches the use case, such as `InformationRetrievalEvaluator` for retrieval or `EmbeddingSimilarityEvaluator` for scored pairs, and compare runs on its metric. - If you have several data shapes, you do not have to pick one loss: a dict of datasets with a matching dict of losses trains each shape with its own objective in the same run. The interview point is the mapping from label shape to objective: pairs without labels mean a ranking loss, graded scores mean an ordering loss, binary labels mean a contrastive loss, and a triplet dataset does not oblige you to use `TripletLoss`.

  • Your duplicate-ticket labels are 1 for duplicate and 0 for not. Which Sentence Transformers losses fit, and is there another route?
    `ContrastiveLoss` or `OnlineContrastiveLoss` take `(anchor, other)` pairs with a 1/0 label directly. Alternatively, keep only the positive pairs and train `MultipleNegativesRankingLoss`, optionally using the 0-labelled partners as an explicit negative column, which turns the task into ranking.
  • What does the scale argument of MultipleNegativesRankingLoss do, and when would you change it?
    The similarities are multiplied by `scale` before the softmax; it is the inverse of a temperature, so the default 20 equals a temperature of 0.05. A higher scale sharpens the focus on the positive. If you switch `similarity_fct` to a dot product, the docstring advises a scale of 1.
  • Can MultipleNegativesRankingLoss use more than one hard negative per anchor?
    Yes. It accepts `(anchor, positive, negative_1, ..., negative_n)` rows, adding every negative column to the candidates for that anchor. The number of columns must be fixed across the dataset; a column holding a variable-length list of negatives is not supported for `SentenceTransformer` training.

  • CosineSimilarityLoss is the right loss for unlabelled question-answer pairs.
  • MultipleNegativesRankingLoss needs an explicit negative in every row.
  • TripletLoss defaults to cosine distance with a small margin like 0.1.
  • Any loss works as long as the dataset has two text columns.
  • A triplet dataset means you have to train with TripletLoss.
open as a page

In Sentence Transformers, how does SentenceTransformerTrainer decide which dataset column is the anchor, which is the positive and which is the label?

level: juniorimportance: should knowfreq 18%

basics

~20 s

By position. A column named label, labels, score or scores becomes the label; every other column is an input in dataset order, so the first is the anchor and the second the positive, whatever they are called.

open as a page

When you run Sentence Transformers' util.mine_hard_negatives over (question, answer) pairs, how do you stop it from mining answers that are actually correct?

level: seniorimportance: should knowfreq 9%

basics

~20 s

Treat the very closest candidates as suspect: skip the top ranks with range_min, require a gap below the positive with relative_margin or absolute_margin, cap max_score, and optionally rescore candidates with a cross_encoder before they are filtered.

open as a page

A Sentence Transformers MultipleNegativesRankingLoss run on support-ticket (question, answer) pairs, where many tickets share one canonical answer, barely beats the base model; what is going wrong, and what do you change?

level: seniorimportance: should knowfreq 14%

basics

~20 s

The in-batch negatives are poisoned: a repeated canonical answer lands in the same batch as its twin and is scored as a wrong answer. Set batch_sampler=BatchSamplers.NO_DUPLICATES, and raise the batch size, with CachedMultipleNegativesRankingLoss if memory runs out.

open as a page

How do you wire Sentence Transformers' InformationRetrievalEvaluator into SentenceTransformerTrainer so the kept checkpoint is the one with the best nDCG@10, not the lowest loss?

level: middleimportance: nice to knowfreq 8%

basics

~10 s

Build InformationRetrievalEvaluator from queries, corpus and relevant_docs dicts with a name, pass it as evaluator=, and set load_best_model_at_end=True with metric_for_best_model naming its key, such as eval_support-dev_cosine_ndcg@10.

open as a page

How do you add MatryoshkaLoss to a Sentence Transformers fine-tune, and what must its matryoshka_dims list contain?

level: middleimportance: nice to knowfreq 5%

basics

~20 s

Wrap the real loss: MatryoshkaLoss(model, base_loss, matryoshka_dims=[...]) applies the base loss to each leading slice of the embedding and sums the results. The list should include the model's full dimension, and no entry may exceed it.

open as a page