skip to content

Sentence Transformers

SentenceTransformer encodes text into dense vectors, CrossEncoder scores query-passage pairs, and both train with the library's own trainers. Interviewers treat it as self-hosted search's workhorse.

on this pageshow

questions

21

In Sentence Transformers, when do you call CrossEncoder.predict and when CrossEncoder.rank, and what does each one return?

level: juniorimportance: must knowfreq 24%

answer

  1. pairs in, scores out
  2. one query against many documents
  3. input order against sorted order
  4. corpus_id, score, optional text
  5. top_k and return_documents on rank

basics

~20 s

CrossEncoder.predict scores a list of (query, passage) pairs and returns one score per pair in input order. CrossEncoder.rank takes one query and a list of documents and returns them sorted by score as corpus_id/score dicts.

solid answer

~40 s

`predict(pairs)` is the general call: you build the pairs yourself, it batches them (`batch_size=32` by default) and returns a numpy `float32` array aligned with your input order, or a tensor with `convert_to_tensor=True`. `rank(query, documents, top_k=None, return_documents=False)` is a convenience for reranking: it pairs the one query with every document, calls `predict` internally, and returns a list of dicts with `corpus_id` (the position in the `documents` list you passed) and `score`, sorted highest first, plus `text` if you ask for it. In Sentence Transformers v6 the `score` values from `rank` are plain Python floats. `rank` only works for single-score models (`num_labels=1`); classifier checkpoints such as NLI models go through `predict`.

code

python · 16 lines
python
from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
query = "How many people live in Berlin?"
passages = [
    "Berlin had a population of 3,520,031 registered inhabitants.",
    "Berlin is well known for its museums.",
]

# predict: you build the pairs, scores come back in input order
scores = model.predict([(query, p) for p in passages])
print(scores)  # numpy float32 array, one score per pair

# rank: one query against many documents, sorted best first
for hit in model.rank(query, passages, top_k=1, return_documents=True):
    print(hit["corpus_id"], hit["score"], hit["text"])

go deeper

for a junior

Recall the two shapes: predict takes pairs and returns scores in your order; rank takes one query plus documents and returns sorted dicts with corpus_id and score.

for a middle

Explain that rank wraps predict, that top_k and return_documents shape its output, and that corpus_id indexes the list you passed rather than any wider corpus.

for a senior

Show you choose predict for multi-label models, mixed query batches and tensor output, and that you know rank's v6 Python-float scores and its num_labels=1 restriction.

for a principal

Frame the choice as API ergonomics against control: rank is the right default for single-query reranking, while predict is the seam for batching heterogeneous pairs across devices.

## Two calls over the same model A **CrossEncoder** in Sentence Transformers reads a query and a passage *together* in one forward pass and outputs a score for that pair. It never produces a vector for a single text, so everything you do with it is pair scoring. The library gives you two entry points: - **`predict(inputs, ...)`** — the general-purpose call. `inputs` is a list of pairs (or one pair). You decide what the pairs are. - **`rank(query, documents, ...)`** — a reranking convenience. You hand over one query and a list of candidate documents; it builds the pairs for you, scores them and sorts the result. `rank` is implemented on top of `predict`: it creates `[[query, doc] for doc in documents]`, calls `predict` on that list, and then sorts. The options `rank` forwards to `predict` — `batch_size`, `activation_fn`, `apply_softmax`, `device`, `show_progress_bar` and the prompt arguments — behave the same way in both calls. ## What each one returns | | `predict` | `rank` | |---|---|---| | Input | list of (a, b) pairs, or a single pair | one query + a list of documents | | Order of output | same order as the input pairs | sorted by score, highest first | | Default type | numpy `float32` array | list of dicts | | Keys / shape | one score per pair (one row of scores per pair for multi-label models) | `corpus_id`, `score`, and `text` when `return_documents=True` | | Truncating the output | slice it yourself | `top_k` (default `None` = return all) | | Models supported | any `num_labels` | only `num_labels=1`, otherwise it raises `ValueError` | A few details worth knowing: 1. **`predict` keeps your order.** Internally it sorts the pairs by length so that batches pad less, then restores the original order before returning. You can zip the result with your input list safely. 2. **A single pair returns a single value.** `model.predict(("q", "p"))` gives one score, not an array of length one. 3. **Output type is controlled on `predict`.** `convert_to_numpy=True` is the default; `convert_to_tensor=True` returns one stacked tensor. In Sentence Transformers v6, `rank` no longer has these switches: its scores are Python floats, and passing `convert_to_numpy` or `convert_to_tensor` to `rank` only logs a deprecation warning. 4. **`corpus_id` in `rank` is a position in the list you passed**, not an id from any wider corpus. If the documents came from an earlier retrieval step, you map back through that step's hits yourself. ## When to use which Use **`rank`** for the common reranking shape: one user query, a short list of candidates from a first-stage retriever, and you want the best few in order. `top_k=10` trims the result and `return_documents=True` saves a lookup when you only need the text. Use **`predict`** when: - the pairs are not one query against many documents — for example, scoring a batch of *different* queries each with its own candidate in one call, or scoring duplicate-question pairs; - the model is a **classifier with several labels**, such as `cross-encoder/nli-deberta-v3-base`, which returns three logits per pair (use `apply_softmax=True` to turn them into probabilities that sum to 1 per pair); - you want an **array or tensor** to feed into further computation rather than a list of dicts. ## Loading the model Both calls hang off the same object: ```python from sentence_transformers import CrossEncoder model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2") scores = model.predict([("How many people live in Berlin?", "Berlin is well known for its museums.")]) top = model.rank("How many people live in Berlin?", passages, top_k=3, return_documents=True) ``` The constructor takes the Hub id (or a local path) and keyword-only options such as `max_length`, `activation_fn`, `num_labels` and `device`. Passing those positionally was deprecated in Sentence Transformers v5.4 and still works with a warning. Larger community checkpoints such as `BAAI/bge-reranker-base` load the same way. ## Common mistakes - Treating `rank`'s `corpus_id` as an id into the full collection rather than into the candidate list. - Calling `rank` on a multi-label classifier and being surprised by the `ValueError`. - Re-sorting `predict` output and losing the alignment with the input pairs. - Expecting `rank` to accept `convert_to_tensor=True` and return tensors in v6 — call `predict` for that. The short version for an interview: **`predict` is "score these pairs, in my order"; `rank` is "sort these documents for this query".**

  • How would you spread CrossEncoder.predict over two GPUs?
    Pass a list of devices, for example `device=["cuda:0", "cuda:1"]`; `predict` then starts a multi-process pool automatically and splits the pairs into chunks. For repeated calls you can create the pool once with `start_multi_process_pool()`, pass it as `pool=`, and stop it with `stop_multi_process_pool()` when done.
  • Why might predict on a three-label NLI cross-encoder need apply_softmax=True?
    A multi-label CrossEncoder uses an identity activation by default, so `predict` returns raw logits per class. `apply_softmax=True` normalises each pair's row so the class scores sum to 1, which you need if you want probabilities rather than just the argmax label.
  • What does batch_size change in a CrossEncoder call?
    Only how many pairs go through the model per forward pass; the default is 32 on both `predict` and `rank`. It trades memory for throughput and does not change the scores. The docs advise benchmarking a few values on your hardware, model and typical input length.

  • rank returns scores in the same order as the input documents
  • rank's corpus_id is the document's id in the whole collection
  • predict returns a list of dicts with corpus_id and score
  • rank works for any CrossEncoder, including multi-label NLI classifiers
  • A CrossEncoder can embed each passage once and reuse the vectors

Assumes Sentence Transformers v6; rank returns Python float scores and its convert_to_numpy / convert_to_tensor arguments are deprecated no-ops since v6.0.

open as a page

In Sentence Transformers, what do model.encode() and model.similarity() return when you embed a list of sentences and compare them?

level: juniorimportance: must knowfreq 40%

basics

~20 s

encode() returns a numpy array with one row per input (a 1-D vector for a single string); similarity() returns a torch tensor of shape [len(a), len(b)] scoring every pair, with cosine as the default function.

open as a page

A Sentence Transformers semantic-search prototype re-encodes the whole corpus on every query; how do you restructure it around util.semantic_search?

level: middleimportance: must knowfreq 25%

basics

~10 s

Encode the corpus once with encode_document() and cache the result; per request encode only the query with encode_query(), call util.semantic_search(query_emb, corpus_emb, top_k=k), and map each hit's corpus_id back to your texts.

open as a page

When picking a Sentence Transformers checkpoint for English passage retrieval from the MTEB leaderboard, which scores and model-card facts should decide it?

level: middleimportance: must knowfreq 30%

basics

~20 s

Read the Retrieval task average for the languages you serve rather than the overall MTEB mean, then the card: parameter count, embedding dimension, max_seq_length against your chunk length, and whether the score assumed query or passage prompts.

open as a page

In Sentence Transformers, which loss fits unlabelled (question, answer) pairs, pairs with a similarity score, and (anchor, positive, negative) triplets, and why?

level: middleimportance: must knowfreq 28%

basics

~10 s

Unlabelled pairs: MultipleNegativesRankingLoss, which uses the rest of the batch as negatives. Scored pairs: CoSENTLoss or CosineSimilarityLoss. Triplets: MultipleNegativesRankingLoss again, with the negative added, or TripletLoss with a tuned margin.

open as a page

In Sentence Transformers, how do you check and change how many tokens a loaded SentenceTransformer reads from each input text?

level: juniorimportance: should knowfreq 22%

basics

~10 s

Read the model's max_seq_length property and assign a new value before encoding. Tokens past it are dropped without an error, and it must stay within the encoder's position-embedding limit, max_position_embeddings.

open as a page

In Sentence Transformers, how does SentenceTransformerTrainer decide which dataset column is the anchor, which is the positive and which is the label?

level: juniorimportance: should knowfreq 18%

basics

~20 s

By position. A column named label, labels, score or scores becomes the label; every other column is an input in dataset order, so the first is the anchor and the second the positive, whatever they are called.

open as a page

After reranking util.semantic_search hits with CrossEncoder.rank, your search returns the wrong passages; what mapping did you miss?

level: middleimportance: should knowfreq 13%

basics

~10 s

CrossEncoder.rank numbers its results by position in the candidate list you passed, while util.semantic_search's corpus_id indexes the full corpus. Map each reranked result back through the hit list: hits[r['corpus_id']]['corpus_id'].

open as a page

Why does CrossEncoder cross-encoder/ms-marco-MiniLM-L6-v2 return scores like 8.6 and -4.3, and how does activation_fn change them?

level: middleimportance: should knowfreq 16%

basics

~10 s

The MS MARCO cross-encoders ship with an identity activation, so predict returns raw logits, roughly -10 to 10. Passing activation_fn=torch.nn.Sigmoid() squashes them into 0 to 1 without changing the ranking order.

open as a page

How does batch_size affect Sentence Transformers' encode() when embedding a few million passages, and how do you choose a value?

level: middleimportance: should knowfreq 12%

basics

~20 s

batch_size (default 32) is how many inputs share one forward pass. Larger batches raise GPU throughput until memory runs out; encode() length-sorts inputs so batches pad little, then restores your order. Benchmark a few sizes on a sample.

open as a page

In Sentence Transformers v6, how do normalize_embeddings and similarity_fn_name work together, and how do you switch a model to dot-product scoring?

level: middleimportance: should knowfreq 15%

basics

~10 s

encode(normalize_embeddings=True) L2-normalizes each vector; similarity_fn_name picks what model.similarity() computes. In v6 switch with model.similarity_fn_name = "dot" or the constructor argument — assigning a function to model.similarity now silently shadows the method.

open as a page

A Sentence Transformers CrossEncoder reranker misses answers near the end of long passages; how does max_seq_length truncate a (query, passage) pair?

level: seniorimportance: should knowfreq 9%

basics

~20 s

A CrossEncoder tokenizes query and passage as one sequence capped at max_seq_length, truncating with longest_first, which trims the longer text, usually the passage tail. Text past the cap is never seen, so split long documents into shorter passages.

open as a page

In Sentence Transformers, a retrieval prototype on an asymmetric checkpoint calls encode() for both queries and passages; what goes wrong, and how do encode_query, encode_document and prompts fix it?

level: seniorimportance: should knowfreq 12%

basics

~20 s

Plain encode() applies only a prompt or prompt_name you pass, or default_prompt_name, so an asymmetric checkpoint embeds queries without its trained prefix. encode_query() and encode_document() pick the model's query and document prompts, and set the routing task, when you pass none.

open as a page

To fit a Sentence Transformers retrieval index into 500 GB, how do you produce int8 or binary embeddings and keep recall with rescoring?

level: seniorimportance: should knowfreq 10%

basics

~20 s

Encode in float32, convert with quantize_embeddings to int8 using ranges calibrated once on a large sample, or to binary, which needs no calibration. Keep queries in float32 and rescore a shortlist of top_k times a multiplier to recover recall.

open as a page

When you run Sentence Transformers' util.mine_hard_negatives over (question, answer) pairs, how do you stop it from mining answers that are actually correct?

level: seniorimportance: should knowfreq 9%

basics

~20 s

Treat the very closest candidates as suspect: skip the top ranks with range_min, require a gap below the positive with relative_margin or absolute_margin, cap max_score, and optionally rescore candidates with a cross_encoder before they are filtered.

open as a page

A Sentence Transformers MultipleNegativesRankingLoss run on support-ticket (question, answer) pairs, where many tickets share one canonical answer, barely beats the base model; what is going wrong, and what do you change?

level: seniorimportance: should knowfreq 14%

basics

~20 s

The in-batch negatives are poisoned: a repeated canonical answer lands in the same batch as its twin and is scored as a wrong answer. Set batch_sampler=BatchSamplers.NO_DUPLICATES, and raise the batch size, with CachedMultipleNegativesRankingLoss if memory runs out.

open as a page

In Sentence Transformers, how does truncate_dim behave when set at load versus at encode, and what does it do to normalized vectors?

level: middleimportance: nice to knowfreq 6%

basics

~20 s

truncate_dim keeps the first N dimensions of each embedding: the constructor value is the default, an encode argument overrides it per call. Truncation happens after the model's Normalize module, so vectors stop being unit length unless normalize_embeddings=True.

open as a page

How do you wire Sentence Transformers' InformationRetrievalEvaluator into SentenceTransformerTrainer so the kept checkpoint is the one with the best nDCG@10, not the lowest loss?

level: middleimportance: nice to knowfreq 8%

basics

~10 s

Build InformationRetrievalEvaluator from queries, corpus and relevant_docs dicts with a name, pass it as evaluator=, and set load_best_model_at_end=True with metric_for_best_model naming its key, such as eval_support-dev_cosine_ndcg@10.

open as a page

How do you add MatryoshkaLoss to a Sentence Transformers fine-tune, and what must its matryoshka_dims list contain?

level: middleimportance: nice to knowfreq 5%

basics

~20 s

Wrap the real loss: MatryoshkaLoss(model, base_loss, matryoshka_dims=[...]) applies the base loss to each leading slice of the embedding and sums the results. The list should include the model's full dimension, and no entry may exceed it.

open as a page

A half-precision CrossEncoder reranker on Sentence Transformers v5 gave several top passages identical scores; why, and what does v6 change?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

In Sentence Transformers v5 a float16 or bfloat16 reranker applied its Sigmoid in half precision, which saturates: strongly relevant passages rounded to the same value near 1.0 and tied. Since v6, predict and rank upcast scores to float32 before the activation.

open as a page

A CPU service loads a Sentence Transformers model with backend='onnx' after you exported an int8 ONNX file, yet latency barely moved; what went wrong, and how do you load it correctly?

level: seniorimportance: nice to knowfreq 7%

basics

~10 s

With backend='onnx' and no file name, Sentence Transformers loads model.onnx or onnx/model.onnx, the float32 export. Name the int8 file with model_kwargs={'file_name': 'onnx/model_qint8_avx512_vnni.onnx'}, and pin revision if it sits in an unmerged pull request.

open as a page