A news feed's post-scoring rule layer has grown to forty rules from four teams - what governance keeps it honest?

principalshould knowfreq 37%

answer

forty conditionals nobody can enumerate
precedence as data, not insertion order
deny beats allow inside a class
owner, reason, scope, expiry
attribution logging plus a rule-off holdout

basics

~20 s

Make precedence a property of the system, not of insertion order; give every rule an owner, a reason and an expiry; measure each rule's cost with holdouts and per-rule attribution logging; and ship and retire a rule through the same ramp and flag a model change gets.

solid answer

~40 s

Four commitments. Order: every rule declares a class - hard eligibility, then must-carry, then suppression, then de-duplication and quotas, then boosts - and conflicts resolve by class with deny beating allow, so two rules never race on insertion order. Provenance: each rule carries an owner, a stated reason, a scope and an expiry date, and expires by default so the layer shrinks unless someone re-justifies it. Measurement: per-rule attribution logging - how many slots this rule moved or removed per thousand requests - plus a small holdout that serves without the rule, with legal and safety rules never eligible for a holdout. Lifecycle: a new rule ramps behind a flag like a model change, and the layer has a rule budget so adding one means retiring one or paying the measured relevance out loud.

Four commitments. **Order**: every rule declares a class - hard eligibility, then must-carry, then suppression, then de-duplication and quotas, then boosts - and conflicts resolve by class with deny beating allow, so two rules never race on insertion order. **Provenance**: each rule carries an owner, a stated reason, a scope and an expiry date, and expires by default so the layer shrinks unless someone re-justifies it. **Measurement**: per-rule attribution logging - how many slots this rule moved or removed per thousand requests - plus a small holdout that serves without the rule, with legal and safety rules never eligible for a holdout. **Lifecycle**: a new rule ramps behind a flag like a model change, and the layer has a rule budget so adding one means retiring one or paying the measured relevance out loud.

code

json · 19 lines

json

{
  "ruleId": "region-licence-block",
  "class": "hard-eligibility",
  "precedence": 10,
  "effect": "deny",
  "owner": "syndication",
  "reason": "content not licensed for delivery outside the publishing region",
  "scope": {
    "surface": "news-feed",
    "appliesWhen": "request.region != item.publishingRegion"
  },
  "expiresAt": "2026-12-31",
  "holdoutEligible": false,
  "rolloutPercent": 100,
  "attribution": {
    "logRemovals": true,
    "logSlotMoves": true
  }
}

go deeper

for a junior

Recall that the rules editing a feed's final list are owned by people and can conflict, so somewhere there has to be a stated order in which they apply.

for a middle

Explain the class ordering and why deny beating allow within a class removes most conflicts without anyone arbitrating case by case.

for a senior

Describe the instruments you would actually build - per-rule attribution logging, a rule-off holdout for eligible rules, and shipping each rule behind a flag with a ramp.

for a principal

Own the trade: a rule budget, expiry by default, and a stated position on how much measured relevance the layer is allowed to spend and who is permitted to spend it.

Forty rules is a system, not a list

At three or four rules the layer is a few conditionals somebody can read. At forty, written by four teams over two years, it has acquired the properties of a system without anyone designing it: an evaluation order nobody chose, an aggregate cost nobody has measured, and a set of interactions nobody can enumerate. The characteristic symptoms:

The ranking stage is blamed for a decline the rule layer caused, because the rules are not in the dashboard and the model is.
Two rules disagree and the winner depends on which was appended last.
A rule added for one event two quarters ago is still firing, and its author has changed teams.
Nobody can answer "what is this layer costing us in relevance" with a number.

Governance here is not process for its own sake. It is the machinery that makes those four questions answerable.

A total order over rule classes

The first commitment is that precedence is a property of the system rather than of the code's shape. Every rule declares a class, and classes evaluate in a fixed order:

Hard eligibility - legal, licence, region, safety. A deny here is final and nothing downstream can overturn it.
Must-carry - editorial pins, contractual placements. Applied only to items that survived step 1.
Suppression - takedowns and per-reader mutes, which remove without being legal absolutes.
De-duplication, quotas and diversity shaping - the edits that change selection among what is left.
Boosts - the only class that rewrites the score, applied last and bounded.

Within a class, deny beats allow. That single rule removes most conflicts, and the ones it does not remove become explicit: two rules in the same class with opposite effects are a design question, surfaced at review rather than discovered in an incident.

What every rule record carries

A rule is a record with required fields, not a branch in a function:

owner - a team, and a rule with no owner is deleted rather than inherited.
reason - why it exists, in a sentence a reviewer can evaluate later.
class and precedence - so the order above is data.
scope - which surface, which regions, which readers.
expiry - a date, defaulting to short. Expiry by default is what makes the layer shrink; renewal is a small, cheap decision, while removal after the fact is a scary one nobody volunteers for.
holdout eligibility - false for legal and safety, true for most of the rest.

Measuring what the layer costs

Two instruments, and you want both:

Per-rule attribution logging. For every request, record which rules fired, which items they moved or removed, and at which slots. This gives cheap, always-on answers: this rule touches four requests in a thousand, that one touches six in ten.
A rule-off holdout. A small slice of traffic served without one eligible rule, so its effect on engagement and on measured relevance is an observed number rather than an argument. Legal, licence and safety rules are never held out - the correct answer for those is that the cost is not the point.

The two answer different questions. Attribution tells you how much a rule acts; the holdout tells you what its acting is worth. A rule that fires constantly and changes nothing measurable is the best retirement candidate in the layer.

Shipping and retiring a rule

The last commitment is that a rule is a change to production behaviour and gets the same treatment as one:

Behind a flag, ramped. A rule that goes straight to all traffic is a deployment without a rollback.
Reviewed by the layer's owner, not only by the requesting team - the reviewer's job is precedence and interaction, which the author is not positioned to see.
On a budget. A stated ceiling on the number of active rules turns "add one more" into a trade instead of an accumulation, and the trade is what forces the retirement conversation that otherwise never happens.
With a documented interaction check against the rules in adjacent classes - particularly quotas against de-duplication, since both defer candidates and together they can empty the pool faster than the over-fetch budget allows.

None of this makes the layer smaller by itself. What it does is make its size, its cost and its authorship visible, which is the precondition for anyone choosing to make it smaller.

## Forty rules is a system, not a list At three or four rules the layer is a few conditionals somebody can read. At forty, written by four teams over two years, it has acquired the properties of a system without anyone designing it: an evaluation order nobody chose, an aggregate cost nobody has measured, and a set of interactions nobody can enumerate. The characteristic symptoms: - The ranking stage is blamed for a decline the rule layer caused, because the rules are not in the dashboard and the model is. - Two rules disagree and the winner depends on which was appended last. - A rule added for one event two quarters ago is still firing, and its author has changed teams. - Nobody can answer "what is this layer costing us in relevance" with a number. Governance here is not process for its own sake. It is the machinery that makes those four questions answerable. ## A total order over rule classes The first commitment is that precedence is a property of the system rather than of the code's shape. Every rule declares a class, and classes evaluate in a fixed order: 1. **Hard eligibility** - legal, licence, region, safety. A deny here is final and nothing downstream can overturn it. 2. **Must-carry** - editorial pins, contractual placements. Applied only to items that survived step 1. 3. **Suppression** - takedowns and per-reader mutes, which remove without being legal absolutes. 4. **De-duplication, quotas and diversity shaping** - the edits that change selection among what is left. 5. **Boosts** - the only class that rewrites the score, applied last and bounded. Within a class, **deny beats allow**. That single rule removes most conflicts, and the ones it does not remove become explicit: two rules in the same class with opposite effects are a design question, surfaced at review rather than discovered in an incident. ## What every rule record carries A rule is a record with required fields, not a branch in a function: - **owner** - a team, and a rule with no owner is deleted rather than inherited. - **reason** - why it exists, in a sentence a reviewer can evaluate later. - **class and precedence** - so the order above is data. - **scope** - which surface, which regions, which readers. - **expiry** - a date, defaulting to short. Expiry by default is what makes the layer shrink; renewal is a small, cheap decision, while removal after the fact is a scary one nobody volunteers for. - **holdout eligibility** - false for legal and safety, true for most of the rest. ## Measuring what the layer costs Two instruments, and you want both: - **Per-rule attribution logging.** For every request, record which rules fired, which items they moved or removed, and at which slots. This gives cheap, always-on answers: this rule touches four requests in a thousand, that one touches six in ten. - **A rule-off holdout.** A small slice of traffic served without one eligible rule, so its effect on engagement and on measured relevance is an observed number rather than an argument. Legal, licence and safety rules are never held out - the correct answer for those is that the cost is not the point. The two answer different questions. Attribution tells you how *much* a rule acts; the holdout tells you what its acting is *worth*. A rule that fires constantly and changes nothing measurable is the best retirement candidate in the layer. ## Shipping and retiring a rule The last commitment is that a rule is a change to production behaviour and gets the same treatment as one: - **Behind a flag, ramped.** A rule that goes straight to all traffic is a deployment without a rollback. - **Reviewed by the layer's owner**, not only by the requesting team - the reviewer's job is precedence and interaction, which the author is not positioned to see. - **On a budget.** A stated ceiling on the number of active rules turns "add one more" into a trade instead of an accumulation, and the trade is what forces the retirement conversation that otherwise never happens. - **With a documented interaction check** against the rules in adjacent classes - particularly quotas against de-duplication, since both defer candidates and together they can empty the pool faster than the over-fetch budget allows. None of this makes the layer smaller by itself. What it does is make its size, its cost and its authorship visible, which is the precondition for anyone choosing to make it smaller.

Which rules must never be put in a holdout, and what do you measure for them instead?
Legal, licence and safety rules, because serving without them is the harm the rule exists to prevent. For those you measure compliance rather than value: how often the rule fires, how long a new entry takes to reach every serving region, and whether any path can bypass it. The question for them is correctness, not cost.
Why does expiry by default do more for the layer than a periodic review?
It inverts who has to act. A periodic review asks someone to justify deleting a rule they did not write, against a vague risk that it still matters - so nothing gets deleted. Expiry asks the owner to justify keeping one they do understand, at a moment when the reason is fresh. The layer then shrinks by inaction rather than growing by it.
How do you tell whether a decline came from the ranking model or from the rule layer?
Per-rule attribution logging joined to the served slates. If the rules fired at the same rate and touched the same slots before and after, the change is upstream in retrieval or scoring; if a rule's firing rate or slot impact moved with the decline, you have your candidate. Without that logging both stories fit the data equally well.

saying these in an interview costs you the question

Resolves rule conflicts by whichever was added most recently
Treats the rule layer as configuration rather than production behaviour
Lets a promotion rule override a legal eligibility deny
Holds out a safety rule to measure what it is worth
Has no per-rule logging and blames the ranker for declines
Adds rules without expiry and reviews them all once a year

A news feed's post-scoring rule layer has grown to forty rules from four teams - what governance keeps it honest?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Forty rules is a system, not a list

A total order over rule classes

What every rule record carries

Measuring what the layer costs

Shipping and retiring a rule

▾common follow-ups3 questions

▾red flags6 flags

Forty rules is a system, not a list

A total order over rule classes

What every rule record carries

Measuring what the layer costs

Shipping and retiring a rule

answer

go deeper