skip to content

How do you sample negatives from unplayed items in an implicit-feedback recommender?

level: seniorimportance: should knowfreq 45%

answer

  1. a few negatives per observed positive
  2. redraw every pass, do not freeze
  3. the distribution matters more than the count
  4. uniform draws land in the tail
  5. popularity draws are hard but punish hits

basics

~20 s

Draw a few unobserved items per observed interaction, often around four, and redraw them each pass. The distribution is the real decision: uniform draws are almost all long tail and too easy, popularity-weighted draws are harder but push popular items down.

solid answer

~50 s

Per observed play, draw `k` unplayed items as negatives — 1 to 10 is the usual range, with around 4 a common default — and redraw them every epoch so the model sees fresh contrasts instead of memorising one fixed set. The distribution matters more than `k`. A uniform draw over a head-heavy catalog picks almost entirely obscure items, which are trivially ranked below the positive and teach the model almost nothing about separating popular items from each other. Drawing in proportion to global popularity, often damped as `popularity ** 0.75`, produces harder negatives and, as a side effect, pushes frequently-played items down. Two hazards: some sampled items are **false negatives** the listener would have loved, and items the user was shown and ignored are much safer, harder negatives if you have impression logs. Evaluate on the full catalog, not on the same sampled negatives.

code

python · 22 lines
python
import random
random.seed(7)

catalog = list(range(1000))                        # tracks 0-99 are the head
plays = [5000 if t < 100 else 3 for t in catalog]  # global play counts
pop_w = [p ** 0.75 for p in plays]                 # damped popularity weights
listened = {12, 340, 871}                          # this listener's positives

def draw(n, weights=None):
    out = []
    while len(out) < n:
        t = (random.choice(catalog) if weights is None
             else random.choices(catalog, weights=weights, k=1)[0])
        if t not in listened:
            out.append(t)
    return out

def head_share(sample):
    return round(sum(1 for t in sample if t < 100) / len(sample), 3)

print(head_share(draw(4000)))        # ~0.10 - uniform negatives are nearly all tail
print(head_share(draw(4000, pop_w))) # ~0.97 - popularity-weighted draws hit the head

go deeper

for a junior

Know the basic recipe: for each observed interaction, draw a few items the user has not interacted with and treat them as negatives, because ranking needs something to rank against.

for a middle

Explain the knobs — how many negatives, resampled or fixed, and from which distribution — and why uniform draws over a head-heavy catalog produce negatives that are too easy to be informative.

for a senior

Demonstrate the tradeoff you have lived: popularity-weighted sampling gives harder negatives and suppresses head items, at the cost of more false negatives, and evaluation must not reuse the sampler.

for a principal

Own the framing that the sampling distribution is a policy decision, not a hyperparameter: it silently sets how much the system favours the head, so it belongs in review alongside the ranking objective rather than buried in training code.

## Why sampling at all Implicit feedback gives you positives only, so training needs contrast. One option is to keep every unobserved cell as a weak negative across the whole matrix. The other, which scales better to very large catalogs and to mini-batch training, is to sample: for each observed interaction, draw a small set of items the user has not interacted with and use them as negatives for that update. ## The three knobs **How many.** The ratio `k` of negatives per positive typically runs from 1 to about 10, with roughly 4 a common working default. More negatives give a lower-variance gradient and usually better ranking, with diminishing returns and linear cost. Tune it; do not inherit it. **Fresh or fixed.** Resampling every epoch exposes the model to many more contrasts and acts as a regulariser. A fixed negative set is cheaper and reproducible but the model can memorise it, and any unlucky draw is baked in for the whole run. **From what distribution.** This is where the real judgement lives. ## Uniform sampling in a head-heavy catalog Suppose a catalog where 100 tracks account for most listening and 900 (or 900,000) are rarely touched. A uniform draw over unplayed items returns tail items almost every time — the head is a tiny slice of the item space. Those negatives are *easy*: the model can separate a hit the user played from an obscure track it has almost no data on, using little more than a popularity signal. Easy negatives produce small gradients and teach nothing about the comparison that actually matters at serving time, which is between plausible candidates near the top of the list. ## Popularity-proportional sampling Drawing negatives in proportion to how often an item appears in the logs — often damped, for example proportional to `count ** 0.75` — flips the composition: negatives are now mostly popular items, which are hard to rank below the positive and therefore informative. There is a second, deliberate effect: because popular items are drawn as negatives far more often, the objective systematically pushes their scores down, which counteracts the head-dominance the positives themselves carry. Overdo it and you suppress genuinely good popular recommendations, so the damping exponent is a dial between "easy negatives, popular-item bias intact" and "hard negatives, popularity actively penalised". ## False negatives Every sampled negative is an assumption. Some fraction of the drawn items are things the listener has simply never encountered and would love. Under uniform sampling these are rare and mild — obscure items, small gradients. Under aggressive popularity sampling they are common and costly, because the items most likely to be a false negative are exactly the popular ones you are drawing most often. Mitigations: exclude items very similar to the user's known positives, exclude anything the user has interacted with in any way, and cap how hard the sampled negatives are allowed to be. ## Harder negatives from exposure data If the product logs what was rendered, items shown to a user and never opened are a much better negative pool than the general unobserved region: the user demonstrably had the chance to act and did not. They are also naturally hard, since the ranker chose to show them in the first place. Blending a portion of exposed-not-opened negatives with sampled unobserved ones is a standard upgrade when impression logs exist. ## Evaluation trap A very common mistake is to evaluate with the same sampling shortcut: rank the held-out positive against a handful of sampled negatives and report the result. Scores from that protocol are inflated and, worse, are not comparable between models, because the difficulty depends entirely on the sampling distribution. Rank the held-out positive against the full candidate set, or against a fixed, documented candidate pool used identically for every model you compare. ## Saying it well Name the three knobs, state the head-heavy catalog problem concretely, and describe the popularity-damping exponent as a dial with a cost on both ends. Finishing with the false-negative hazard and the evaluation trap shows you have actually run this rather than read about it.

  • What goes wrong if you fix the negative set once instead of resampling?
    The model sees the same contrasts repeatedly and can memorise them, so the effective number of training comparisons collapses and any unlucky draw is baked in for the whole run. Resampling each epoch multiplies the contrasts seen for the same positives and acts as a regulariser. The cost is reproducibility, which you buy back with a seeded sampler rather than a frozen set.
  • Why is evaluating against sampled negatives a problem?
    Difficulty depends entirely on the sampling distribution, so the number reports your sampler as much as your model, and two models evaluated with different samplers are not comparable. Ranking a held-out positive against a hundred easy tail items looks excellent and says nothing about serving, where the competition is the top few hundred candidates. Rank against the full catalog or a fixed documented pool.
  • How do you limit damage from false negatives?
    Exclude everything the user has touched under any event type, not just the target event, and exclude items very close to their known positives. Cap sampling hardness rather than always mining the hardest available items, and if you have impression data, prefer exposed-not-opened negatives, where the user demonstrably declined. Accept that some false negatives remain; the aim is to bound them, not to eliminate them.

saying these in an interview costs you the question

  • Samples uniformly and never checks what got drawn
  • Freezes one negative set for the whole training run
  • Assumes every sampled unplayed item is a true dislike
  • Reports metrics computed against sampled negatives
  • Treats the negatives-per-positive ratio as a fixed constant

context

resolving links…