Why does a promotion and refund abuse decision layer keep hand-written rules after a model score is added?
answer
- known patterns against the unenumerated rest
- shippable in hours, no labels needed
- some lines are policy, not prediction
- a decline needs a readable reason
- the score orders the grey band
basics
~20 sRules cover what is already known, instantly and deterministically: a promo-code exploit seen this week can be blocked today, and policy that must never bend is written rather than learned. The model covers the unenumerated remainder.
solid answer
~50 sRules keep three jobs a fitted score cannot do. First, response time: a newly observed promo-redemption pattern becomes an enabled rule the same day, while learning it needs labelled examples that do not exist yet. Second, policy that is a decision rather than a prediction - 'an account opened under 24 hours ago cannot refund to a payment instrument other than the one charged' is a written commitment, and a score that allows it 8% of the time is a policy breach, not a modelling error. Third, explainability: a declined refund needs a named reason a support agent can read back. The model earns the other half - it generalises to combinations nobody enumerated, and it orders the grey band instead of answering yes or no. The design question is never rules-or-model; it is how the two compose into one emitted action.
go deeper
Recall that rules and the score do different jobs: rules encode what is already known and must hold exactly, the score covers combinations nobody enumerated. Be able to give one example of each.
Explain the mechanics behind the split - why a pattern seen yesterday cannot be learned yet, why a cliff at a rule's constant behaves differently from a continuous score, and what decides whether a new signal becomes a rule or a feature.
Show you have operated the consequence: rules accumulate, so keeping them means owning versions, expiry and a decision record. Demonstrate that you know which commitments must never ride on a probabilistic score.
Frame it as an organisational trade: rules are fast because anyone can author them, and that same property is what turns an unmanaged rule set into a policy nobody can state. The cost being traded is maintenance and auditability, not accuracy.
## The setting On a consumer commerce account, promotion and refund abuse looks concrete: someone opens accounts to redeem a first-order promo code again and again, or requests refunds on delivered goods at a rate no ordinary customer reaches. Behind the *apply promo* and *issue refund* endpoints sits a **decision layer** that must emit exactly one action per request - allow, challenge, decline, or route to a human case review - and it holds two very different instruments. A **rule** is a condition a person wrote: `account_age_hours < 24 AND refunds_requested_30d >= 3 -> decline`. It is exact, readable, and live the moment it is enabled. A **model score** is a fitted estimate of risk over many inputs at once: it generalises to combinations nobody wrote down, and it returns a continuous number rather than a verdict. Most real systems in this archetype grew rules first and added the score underneath them later, which is why the question is asked this way round. ## The three jobs a rule keeps 1. **Response measured in hours.** A pattern first seen yesterday has almost no labelled examples, so there is nothing for a fitting procedure to latch onto. A rule needs one analyst's observation and a condition, not a labelled population. The gap between 'we can describe it' and 'we have enough of it to learn it' is exactly where rules live. 2. **Policy that is a decision, not a prediction.** Some lines are commitments - contractual, regulatory, or simply promised to the finance team. A commitment that holds 92% of the time is not a commitment. A rule expresses it as a condition that either matched or did not; a score expresses it as a tendency. 3. **An explanation that survives the conversation.** When a customer disputes a denied refund, the agent needs a specific reason. A rule supplies one for free: it is a named condition with a version. A score supplies a number, and a number is not a reason anybody can act on. ## What the model contributes - **Coverage of the unenumerated.** Abuse that is only visible as a *combination* - a middling account age, a middling redemption count, a device shared with two other accounts, none of them individually alarming - is precisely what no one writes a rule for. - **Ordering, not just verdicts.** A score turns a pile of suspect requests into a ranked list, which is what makes a human review queue useful at all. Where the cut between bands is placed is a separate design decision. - **Graceful behaviour near a constant.** A rule's condition is a cliff: a request just under `refunds_requested_30d >= 3` is treated exactly like a request from a first-time customer. The score moves continuously across that boundary. - **One number that composes.** Downstream policy can reason about a single risk score; it cannot reason about forty booleans without doing the composition itself. ## Where each one fails alone | Situation | Rules only | Model only | |---|---|---| | Pattern first seen today | Covered within hours | Not learnable yet - no labelled examples | | Unenumerated combination of weak signals | Missed entirely | Its core strength | | Explaining a specific decline | A named condition, for free | A band and contributing features at best | | A guarantee the business has committed to | Holds exactly | Holds most of the time | | Maintenance | Grows without bound, needs lifecycle | Refit, but the set of inputs stays bounded | The honest reading of that table is that the two failure columns barely overlap, which is why the archetype ends up with both rather than with a winner. ## What this implies for the design Every new signal that arrives needs a **home decision**: does it become a rule, or an input to the score? A sharp, newly observed, low-volume pattern belongs in a rule. A signal that has recurred for months with real volume is usually better as a feature, where its weight is fitted against everything else rather than guessed by whoever wrote the condition. And because rules accumulate, keeping them is a commitment to maintaining them: an owner, a version, a review date, and a decision record naming which rule fired. A decision layer that keeps rules without that discipline ends up with a policy nobody can state, which is a worse outcome than either instrument alone.
- What kind of signal is better added as a model feature than as a new rule?One that has recurred for months and carries real volume. Its weight can then be fitted against every other input rather than guessed by whoever wrote the condition, and it degrades gracefully near its boundary instead of behaving as a cliff. A pattern first seen this week is the opposite case: too few examples to move a fit, but perfectly expressible as a condition.
- Why does retraining more often not remove the need for rules?Learning a pattern requires labelled examples of it, and a pattern first observed yesterday has almost none regardless of how often the fit runs. A rule needs one analyst's observation instead of a labelled population. Frequent refitting also does nothing for the second job rules hold - expressing a commitment that must hold on every request, not most of them.
saying these in an interview costs you the question
- Rules are technical debt that a good model eventually makes unnecessary
- A model will respect any policy if you give it the matching feature
- A new abuse pattern has to wait for the next model refit
- A risk score can be shown to a customer as the reason for a decline
- Every new signal should become a rule, since rules ship fastest