How do you build pairwise ranking training data from result lists of 500 candidates each?

## Where pairs come from Pairwise training data is *derived* data. You do not collect pairs; you collect lists with labels and then decide which pairs to manufacture from them. Every one of those decisions changes what the model learns, which is why this is a design step rather than a preprocessing detail. Start with the unit. A query — one search, one feed request, one user session on a surface — produces a candidate list and a set of labels over it. On a hotel search that might be 500 candidates of which a few were clicked and one was booked, with booked outranking clicked outranking ignored. That list, and only that list, is the universe inside which pairs are legitimate. ## Rule one: pairs live inside one list A pair `(a, b)` asserts *a should be shown above b*. That assertion is only ever acted on when `a` and `b` compete for slots in the same response. If you build a pair from a hotel in a Paris search and a hotel in a Tokyo search, you are asking the model to order two items that will never appear together, and to do so you force it to make its scores comparable across queries — an extra burden that no served ordering benefits from, and one that consumes model capacity. Worse, most useful ranking features are query-relative (how well this hotel matches *these* dates, *this* price filter), so cross-query comparisons are not even well defined. The same rule shows up at evaluation and at split time: keep a query's items together as a group. Splitting one list's candidates across a training and a validation set lets the model see part of the very list it is being scored on. ## Rule two: the count is quadratic A list of `n` candidates has `n * (n - 1) / 2` unordered pairs. For `n = 500` that is 124,750 — from a single query. Ten thousand queries would be over a billion pairs. Two things follow: naive enumeration is not an option, and pair count is not proportional to data volume, so a modest increase in candidate-set depth explodes the training set. The first and cheapest reduction is to drop pairs that carry no preference. If two items share a relevance label — both booked, both ignored — there is no preferred order to encode, and implementations normally exclude them. On a list with 5 relevant items and 495 irrelevant, only the 5 * 495 = 2,475 mixed pairs remain: a fifty-fold cut before any sampling at all. With graded labels the arithmetic is the same idea applied level by level: every pair whose two items sit at different grades is a candidate, and pairs within a grade are dropped. ## Rule three: choose the pairs that teach After the label filter you still have too many, and they are not equally informative: - **Gap size.** A booked hotel against an ignored one states a stronger preference than a clicked one against an ignored one. Weighting by the label gap tells the model which distinctions to prioritise. - **Current model error.** Pairs the model already orders confidently correctly contribute almost nothing to the gradient of a sigmoid-based pairwise loss, and pairs it gets wrong contribute the most. Some setups make this explicit by mining hard pairs; the loss's own shape does a weaker version of it automatically. - **Depth.** An inversion between the top two results matters enormously; one between ranks 400 and 401 does not. A plain pairwise loss is blind to this, and the standard remedy is to weight each pair's gradient by how much swapping those two items would move the list-level metric. ## Rule four: normalise across queries If you simply sum the loss over all surviving pairs, a query with 500 candidates and 20 relevant items contributes orders of magnitude more terms than a query with 30 candidates and 1 relevant item. Training then optimises for the big lists, which are usually the broad, ambiguous queries, at the expense of the sharp ones. The fixes are to divide each query's pair loss by its pair count, or to sample a fixed budget of pairs per query, so every query contributes comparably. Which is right depends on whether you believe a big list is genuinely more important or merely bigger. ## Sanity checks before you train Count the pairs per query and look at the distribution, not the mean — a handful of enormous lists is a common surprise. Verify that no pair spans two query identifiers; this is easy to get wrong when data is shuffled before grouping. Check that queries with no positive and queries with no negative produce zero pairs and are dropped rather than silently contributing nothing while still counting toward your dataset size. And confirm that your per-query normalisation actually changes the loss when list lengths differ — if it does not, it is not wired in.