What does a learning-to-rank re-ranker require that hand-tuned relevance boosts do not?

## The two-stage shape Learning-to-rank is not a replacement for retrieval; it is a re-ranking layer on top of it. Stage one is a cheap, recall-oriented retrieval — a lexical query, a vector query, or both — that returns the top N candidates, typically a few hundred to a couple of thousand. Stage two runs a model over those N candidates and reorders them. The split exists because per-pair model scoring costs orders of magnitude more than an inverted-index scan, and because you only need good ordering at the top. The immediate consequence is a **recall ceiling**: a document the first stage did not return can never be ranked, no matter how good the model is. Measure first-stage recall at N as its own number. If the ideal answer is outside the candidate set for 8% of queries, no re-ranker can fix those 8%, and the right investment is a better candidate generator or a larger N, not more model capacity. ## What you need that a boost config does not **Labels.** A model needs graded examples: for a given query, which documents are perfect, acceptable, or wrong. Two sources exist. Human judgments are expensive and slow but unbiased with respect to your current ranking. Click logs are free and plentiful but confounded — clicks reflect where a document was shown as much as whether it was good — so they need position-bias correction before they become usable labels. Most teams use both: clicks for volume, human judgments for calibration and for evaluation. **A feature pipeline.** Features come in three families: - *Query features*: length, detected intent, language, whether it looks like a navigational query. - *Document features*: age, popularity, quality score, completeness, stock status. - *Query–document features*: the lexical relevance score per field, term coverage, phrase proximity, vector similarity, category match. Every one of those must be computed identically during training and at serving time. **Train/serve skew** — a feature computed from a nightly batch offline and from a live counter online, or a normalisation applied in one place and not the other — is the single most common way a re-ranker that looked excellent offline does nothing in production. The defence is one code path shared by both, plus a check that logs live feature vectors and compares their distribution against the training set. **A latency budget.** Scoring N candidates is linear in N. A gradient-boosted tree ensemble over fifty features can handle a thousand candidates in single-digit milliseconds; a transformer cross-encoder that reads the query and document text together is far more expensive and typically re-ranks tens of documents, not thousands. Choose N and the model family together, against the latency budget, not separately. **A retraining loop.** The corpus changes, the query mix changes, and — critically — your own ranking changes what users see and therefore what the next round of click labels looks like. A model trained once and left alone degrades. Plan for scheduled retraining and for monitoring feature distributions, not just output metrics. ## Model families in one paragraph *Pointwise* approaches predict a relevance score per document independently and are the easiest to train but optimise the wrong thing — absolute score accuracy rather than order. *Pairwise* approaches learn which of two documents should rank higher and match the task better. *Listwise* approaches optimise a ranking objective over the whole result list directly and are what the well-known gradient-boosted implementations do. In practice gradient-boosted decision tree ensembles remain a very strong default for tabular ranking features: fast, robust to feature scaling, and interpretable enough to debug. Neural cross-encoders win on pure text relevance and cost far more per document. ## What you give up Hand-tuned boosts are legible. When a stakeholder asks why a document ranks third, you can point at four numbers. With a model, the honest answer is a feature attribution, which is harder to explain and harder to act on. You also lose the ability to make a targeted fix quickly: with boosts, you change a weight; with a model, you fix the training data and retrain. Keep a curation or pinning mechanism outside the model for the handful of cases that need a deterministic answer. ## When it is not worth it Learning-to-rank pays off with high query volume, a large catalogue, and a team able to maintain a feature pipeline. It does not pay off on a low-traffic site (not enough clicks to learn from), a small catalogue (few enough documents that curation is cheaper), or without a judgment budget (no way to evaluate whether the model helped). In those cases a well-built analyzer chain, sane field weights, and two or three capped signals will beat an untended model — and, more importantly, will still work in a year. ## Where personalisation fits User-specific features — affinity with a category, prior interactions, session context — belong in the re-ranking stage, applied to the top-k. Keeping them out of first-stage retrieval means the candidate set stays shared and cacheable, the blast radius of a personalisation bug is bounded to reordering, and you can fall back cleanly to the impersonal ranking for cold or anonymous users. It also keeps the ranking reproducible enough to debug, which fully personalised retrieval is not.