How does graded relevance change a search-ranking task compared with binary relevant-or-not labels?

## Two different specifications of the same task A binary relevance scheme says each result either belongs in the answer set or does not. A graded scheme assigns a level — a common four-level ladder is **perfect / good / marginal / bad** — and that small change rewrites the task. ### What the model is asked to produce Under binary labels, the task reads as *retrieve the relevant ones*. Any order among the relevant items is equally acceptable, because the labels contain no information distinguishing them. Under graded labels the task reads as *order the results so that higher grades appear before lower ones*. An ordering that places a marginal result above a perfect one is now a defect the labels can describe. This is why graded relevance is the natural home of ranking framing: the label set itself encodes an ordering, so "correct" is a property of the sequence. ### The labelled unit is the pair, not the item A subtle framing point that interviewers probe: a grade attaches to a **(query, document) pair**. A returns-policy page is *perfect* for "how do I return an item" and *bad* for "track my order". Nothing about the document alone determines its grade. Consequences: - Features must describe the match, not just the document — otherwise the model can only learn a global quality prior for pages. - Splitting data for evaluation should generally be done **by query**, so that judgements for the same query do not straddle training and evaluation. - The number of labels you need scales with queries times results judged per query, not with documents. ### The guideline becomes the spec With two labels, judges argue over one boundary. With four, they argue over three, and the interior boundaries are the fuzzy ones. Practical effects: - **Agreement falls as grades multiply.** Distinguishing bad from everything else is easy; distinguishing good from marginal is a judgement call, and disagreement there is not noise to be averaged away — it means the guideline is under-specified. - **Grades are ordinal, not numeric.** Perfect is better than good, but it is not "twice as good". Any mapping of grades to numbers is a modelling choice you are making, not something the labels handed you. - **The grade distribution is usually skewed.** Most judged results for a head query are decent; the perfect grade is rare. That rarity is what makes the top of the list hard. ### Collapsing grades Teams often collapse the ladder — for example treating perfect and good as positive and the rest as negative — so they can train a simpler binary model. That is legitimate, but it is a deliberate loss: - Every ordering that keeps positives above negatives becomes equally good, so the model has no reason to lift perfect results above merely good ones. - Where you draw the collapse line silently redefines the product. Folding *marginal* into the positive side tells the model that a barely acceptable result belongs at the top. - The evaluation must match the framing you shipped. If the product promise is "the best answer first", a measure that only counts hits cannot see whether you kept that promise. ### When binary is the honest choice Graded relevance is not automatically better. Prefer binary when the downstream use really is set-shaped (does this result belong in a fixed answer box at all?), when you cannot afford to write and enforce a four-level guideline, or when judge agreement on the interior grades is so low that the extra levels are mostly noise. A clean binary label set beats a graded one that three judges fill in three different ways. ### Where graded labels come from Two sources, with different failure modes. **Human judges** following a guideline give you grades that reflect the product's intent, at a cost per judgement and with the agreement problem above. **Behavioural signals** such as clicks or dwell are cheap and plentiful, but they are observations of what users did with the ordering you already showed them, so they carry the incumbent ranking's bias and are not a neutral measurement of relevance. Many teams use judged grades as the specification and behaviour as a volume supplement, being explicit about which is which. ### Interview answer shape Say what changes in the *task*: the target becomes an ordering rather than a set; the unit of labelling is the query-document pair; the guideline that defines the middle grades is now part of the product spec; and collapsing grades is a real information loss you should name out loud rather than perform silently.