In Haystack, why must TransformersSimilarityRanker be warmed up before it ranks?

## Construction is cheap, warm-up is expensive Haystack draws a hard line between a component's *configuration* and its *resources*. Constructors record settings — model name, device, `top_k`, prefixes, thresholds — and nothing more. That property is what makes `Pipeline.to_dict()` and YAML serialisation meaningful: the artefact describes a pipeline in terms anyone can reload, and building it does not require the model weights to be present. `warm_up()` is the second phase. For `TransformersSimilarityRanker` it resolves and downloads the cross-encoder if it is not cached, instantiates it, and places it on the selected device. This can take seconds and hundreds of megabytes to gigabytes of memory. The same pattern applies to the other model-backed components — embedders and the diversity ranker warm up the same way. ## Who calls it Inside a pipeline you generally do not. `Pipeline.run()` ensures components are warmed before they execute, and `Pipeline.warm_up()` lets you force it at a time of your choosing. Outside a pipeline — in a unit test, a notebook, or a component you drive by hand — you must call `warm_up()` yourself, and forgetting it produces a runtime error rather than silent misbehaviour, which is the right trade. ```python ranker = TransformersSimilarityRanker(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_k=5) ranker.warm_up() result = ranker.run(query="...", documents=docs) ``` ## The operational consequences **Cold start.** If the first HTTP request into your service is what triggers warm-up, that request absorbs the model download and load. Warm the pipeline during application startup, before the process reports ready, so the p95 of real traffic never includes it. In a container, bake the model into the image or a mounted cache so warm-up is a load rather than a network fetch — otherwise a registry outage becomes an outage for you. **Memory per worker.** Warm-up loads the model into the process. Four Gunicorn workers means four copies. This is the usual cause of "it ran fine locally and OOM-ed in production", and it is the reason people move the ranker behind a shared inference service once traffic justifies it. **Latency per request.** A cross-encoder scores every query-document pair it is given, so its cost scales with how many documents the retriever handed it — not with `top_k`, which only trims the output. Widening retrieval depth to improve recall makes the ranker, not the retriever, the expensive component. ## What ranking does to the documents Three behaviours are worth stating precisely because they surprise people: 1. **`score` is overwritten.** The returned documents carry the ranker model's relevance score, not the retriever's or the joiner's. Any threshold you tuned upstream is meaningless afterwards. 2. **`top_k` truncates.** Set on the constructor or per run, it limits the returned list; documents beyond it are dropped, not merely reordered. 3. **`score_threshold`** drops documents scoring below the cut. That is how you get an empty result set from a ranker, and your pipeline must handle "nothing was relevant" rather than sending an empty context to the generator with a prompt that assumes documents exist. The ranker also supports `query_prefix`/`document_prefix` and `meta_fields_to_embed`, which let you fold metadata such as a title into the text the model actually scores — often a bigger quality win than swapping models. ## A version note The Haystack 2.x line introduced `SentenceTransformersSimilarityRanker` as the successor to `TransformersSimilarityRanker`; both are cross-encoder rankers with the same warm-up contract and the same `top_k`/`score_threshold` semantics, so pin the class name against the Haystack version you are running rather than assuming from a tutorial.