How do you train a search ranker when the target metric NDCG has zero gradient?

## Why ranking metrics have no gradient A ranking metric such as NDCG is computed by sorting the candidate items by their predicted scores, then summing each item's gain discounted by its **position**. The scores appear nowhere in that sum except through the sort. Perturb a score slightly and, almost always, the order is unchanged and the metric is unchanged; perturb it past a neighbour and the metric jumps. So NDCG, like accuracy, is piecewise constant in the model parameters — zero gradient almost everywhere, discontinuous at the swaps. Mean average precision and mean reciprocal rank have exactly the same shape for the same reason. ## Three families of surrogate **Pointwise.** Regress each item's relevance label directly, treating ranking as regression or classification per item. It is simple and trains on independent examples, but it optimises absolute score accuracy rather than order, and it spends capacity getting the exact grade right for items whose position is never in doubt. It also does not know that being wrong between the top two items matters more than being wrong deep in the tail. **Pairwise.** This is the workhorse. Form pairs `(i, j)` where `i` should be ranked above `j`, and penalise the model whenever the score difference `d = s_i - s_j` is not comfortably positive. The logistic form is `L_pair = log(1 + exp(-d))` which is smooth, positive everywhere, and decreasing in `d`. Its derivative with respect to `d` is `-1 / (1 + exp(d))` — near `-1` when the pair is badly inverted, near `0` when it is already well ordered. Backpropagating that pushes `s_i` up and `s_j` down. Because the loss depends only on the difference, the model is free to place the absolute scores anywhere, which is precisely right for a metric that only reads order. A margin form, `max(0, m - d)`, encodes the same preference with a hard cushion. **Listwise.** Score the whole candidate list at once and define a loss over the list — for example a softmax over the list's scores compared against the distribution implied by the relevance labels. Listwise objectives see the competition among all candidates directly and can encode top-heaviness naturally, at the cost of more complex batching, since the training unit is now a query with all its candidates. ## Where the pairs come from With editorial relevance grades, pairs are any two items with different grades. With production logs you usually have clicks instead, and the standard construction is a **clicked document paired with a skipped one that was ranked above it**. The 'ranked above' condition matters: if a document was never shown near the top, the user probably never examined it, and the absence of a click on it says nothing about relevance. Pairing only against documents the user plausibly saw and passed over is what makes the label a preference rather than an artefact of position. That is also the central caveat: click data is confounded by position. Items shown higher get clicked more regardless of quality, so a ranker trained naively on clicks learns to reproduce the previous ranker's ordering. Handling that — reweighting pairs by an examination estimate, or injecting randomisation into the served ranking to collect unbiased comparisons — is a whole discipline in itself, and knowing it exists is the difference between a candidate who has read about ranking and one who has shipped it. ## Closing the gap back to the metric A plain pairwise loss counts inversions, and counting inversions is not NDCG. NDCG discounts by position, so a swap between ranks 1 and 2 changes it far more than a swap between 40 and 41, while the pairwise loss values both identically. The influential fix is to weight each pair's gradient by `|delta NDCG|` — the change in the metric that would result from swapping exactly those two items, holding everything else fixed. This is the LambdaRank construction, and its notable property is that it specifies the *gradient* directly rather than deriving it from a loss function that is written down first. It works well in practice and pulls the training signal back toward the top of the list, where the metric actually lives. ## What to say when asked Lead with the diagnosis — the metric is a function of a sort, so it is flat — then name the change of unit from item to pair as the core move, then the logistic-on-the-difference form, then the top-heaviness gap and the delta-metric weighting that closes it. If the interviewer is coming from search or recommendations, expect the conversation to turn immediately to where the preference labels came from, because that, and not the loss algebra, is where real ranking systems go wrong.