When picking a Sentence Transformers checkpoint for English passage retrieval from the MTEB leaderboard, which scores and model-card facts should decide it?
answer
- a column, not the headline
- task type matching your workload
- size and dimension for your hardware
- max_seq_length against chunk length
- prompts the published score assumed
basics
~20 sRead the Retrieval task average for the languages you serve rather than the overall MTEB mean, then the card: parameter count, embedding dimension, max_seq_length against your chunk length, and whether the score assumed query or passage prompts.
solid answer
~40 sI start from the Retrieval column for my languages, because the overall MTEB average blends classification, clustering and STS tasks my search never runs. Then I filter by cost: `sentence-transformers/all-MiniLM-L6-v2` is 22.7M parameters, 384 dimensions and a 256-token `max_seq_length`; `sentence-transformers/all-mpnet-base-v2` gives better quality and 768 dimensions with 384 tokens, but runs about five times slower; `BAAI/bge-base-en-v1.5` is 109M parameters with 512 tokens. The card also says whether the score was produced with prompts: e5 checkpoints want `"query: "` and `"passage: "`, bge an instruction on queries. If `model.prompts` does not carry them, the leaderboard number will not reproduce. The board builds the shortlist; a test on my own queries makes the call.
go deeper
Know that MTEB scores task types separately and that all-MiniLM-L6-v2 and all-mpnet-base-v2 are the classic small and quality baselines.
Explain why the Retrieval column beats the overall mean for search, and read max_seq_length, dimension and prompts off the card or the loaded model.
Show how a missing prefix or a short max_seq_length makes a well-ranked model underperform, and how mteb.get_model reproduces a published score.
Frame the board as a cost-filtered shortlist feeding an evaluation on your own queries, and own the trade between a point of score and CPU cost.
## What the MTEB leaderboard ranks The **Massive Text Embedding Benchmark (MTEB)** scores embedding models on many tasks grouped into task types: **retrieval**, **reranking**, **classification**, **clustering**, **semantic textual similarity (STS)** and more, in many languages. Benchmarks such as `MTEB(eng, v2)` bundle a fixed set of tasks, and the leaderboard Space shows each model's per-task-type averages next to an overall mean. The overall mean is the number people quote and the one that misleads most. A search or RAG feature only ever runs one kind of task: a short query against longer passages. A model can climb the overall table on classification and clustering while sitting lower on retrieval. So the reading order is: 1. Pick the benchmark whose **languages** match your traffic (English-only, or a multilingual one if you serve several languages). 2. Sort by the **Retrieval** task-type average, not the overall mean. 3. Drop models you cannot afford to run; the Sentence Transformers docs explicitly advise filtering away large models that need excessive hardware. 4. Treat what remains as a shortlist, not a verdict. ## What the model card adds The board gives a score; the card and the loaded model give the operating facts. These are the reference points interviewers expect you to know: | Checkpoint | What the docs report | Prompts | |---|---|---| | `sentence-transformers/all-MiniLM-L6-v2` | 22.7M parameters, 384 dimensions, `max_seq_length` 256, mean pooling then `Normalize` | none | | `sentence-transformers/all-mpnet-base-v2` | best quality of the `all-*` family, 768 dimensions, 384 tokens, about 5x slower than MiniLM | none | | `BAAI/bge-base-en-v1.5` | 109M parameters, 512 tokens | retrieval instruction on queries | | `intfloat/multilingual-e5-large` | multilingual | `"query: "` and `"passage: "` | Facts the card may omit you can read from the loaded object: ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-base-en-v1.5") print(model.max_seq_length) # tokens read per input print(model.get_embedding_dimension()) # vector size you will store print(model.prompts) # prompts saved in config_sentence_transformers.json ``` - **`max_seq_length`** must cover your typical chunk; a 256-token model embeds only the start of a 600-token chunk. - **Embedding dimension** multiplies straight into index size and search cost. - **Parameter count** decides CPU latency far more than a point of retrieval score. - **Prompts**: some checkpoints were trained with a prefix on queries, passages or both. The `prompts` dictionary in `config_sentence_transformers.json` carries them when the author saved them; if it is empty but the card asks for a prefix, pass `prompts=` to the constructor yourself. ## Reproducing a published number A leaderboard score is only meaningful for your deployment if you run the model the way it was scored. The MTEB section of the Sentence Transformers docs warns that results vary with **normalization, quantization and prompts**, and recommends loading an existing model through `mteb.get_model(name)` so it runs as its authors registered it. To re-run a task yourself: - `mteb.get_tasks(task_types=["Retrieval"], languages=["eng"])` or `mteb.get_benchmark("MTEB(eng, v2)")` selects tasks; - `mteb.evaluate(model, tasks, encode_kwargs={"batch_size": 64})` runs them and passes arguments through to `encode`; - MTEB is meant for evaluation after training, not as a training-time evaluator. If your own run of an e5 checkpoint scores far below its row, missing prefixes are the first suspect. ## A worked shortlist Suppose an English-only help-centre search runs on CPU, and the chunker produces passages of about 300 tokens. Reading the board and the cards gives a quick first cut: 1. `all-MiniLM-L6-v2` is the cheapest option, but its 256-token limit drops the tail of most chunks, so either the chunker changes or the model is out. 2. `all-mpnet-base-v2` covers 384 tokens and is the quality baseline of its family, at about five times MiniLM's cost. 3. `BAAI/bge-base-en-v1.5` covers 512 tokens at 109M parameters, and needs its query instruction configured. Three candidates survive with known costs and known configuration needs. Nothing about that list says which will retrieve best on help-centre questions, which is why it is a shortlist. ## What the board cannot tell you - **Your domain.** Public retrieval sets are not your support tickets or contracts; a model's rank does not transfer automatically. - **Your hardware.** The board does not run on your CPU at your batch size. - **Your chunking.** A model with a short `max_seq_length` can rank well on short-passage tasks and still lose the tail of your chunks. So the board narrows the field to two or three candidates that fit your languages, budget and chunk length, and an evaluation on your own labelled queries picks the winner. How to run that head-to-head comparison is a separate retrieval-evaluation subject; the skill here is reading the board and the card well enough that the shortlist is right.
- The card for an e5 checkpoint asks for query and passage prefixes, but model.prompts is empty after loading. What do you do?Pass `prompts={"query": "query: ", "passage": "passage: "}` to the `SentenceTransformer` constructor, or assign `model.prompts`. `encode_query` then applies the `query` prompt and `encode_document` picks up `passage` automatically. Calling `save_pretrained` writes them into `config_sentence_transformers.json`, so later loads need no code.
- How do you reproduce a model's published MTEB retrieval score yourself?Install `mteb`, select the same tasks with `mteb.get_tasks` or `mteb.get_benchmark`, and run `mteb.evaluate`. For an existing model the MTEB team recommends `mteb.get_model(name)` over a bare `SentenceTransformer`, because normalization, quantization and prompts change scores and `get_model` loads the model as its authors registered it. `encode_kwargs` passes `batch_size` or `normalize_embeddings` through to `encode`.
- Picks the checkpoint with the highest overall MTEB average for a search feature
- Assumes every checkpoint reads 512 tokens whatever its max_seq_length says
- Runs an e5 checkpoint without its query and passage prefixes and expects the board's score
- Treats a top leaderboard rank as proof the model suits their own domain
- Ignores parameter count and dimension when the service runs on CPU