Why does a fixed 500-downloads-a-day rule miss a user whose SaaS downloads tripled?

## Why one number cannot work A count-over-window rule fires when some entity exceeds a count within a period — downloads per user per day, failed authentications per account per hour, records read per service account per week. The tempting implementation is one threshold for everyone. The trouble is that the threshold must be high enough not to fire on legitimate heavy users, and in every real estate the heaviest legitimate user is one or two orders of magnitude above the median. A backup or reporting identity pulling hundreds of documents daily forces the number up, and once it is up, the entire population of ordinary users has enormous unmonitored headroom underneath it. That headroom is the whole attack surface of the rule. Slow collection through a sanctioned client is specifically shaped to live there: the account is legitimate, the client is approved, the destination is the normal one, and the only anomalous thing is the *rate relative to that account*. Against a global threshold there is nothing to detect. Worse, a global threshold is a published boundary — anyone who can observe whether alerts fire can find the edge and stay below it. ## Deriving a threshold instead of choosing one "Tuned" means the number came from data, not from a round figure someone liked. The method is: 1. Take a representative slice of history — long enough to include the estate's cycles, and free of known incident activity. 2. Compute the per-entity distribution of the count you intend to alert on: median, p95, p99, maximum. 3. Pick the candidate threshold and *backtest* it: how many alerts would this rule have produced over that period, and on which entities? 4. Sample the would-be alerts and label them. The output you care about is the daily volume an analyst would inherit and the share of it that is worth working. 5. Record the number with its derivation and a review date, because the estate drifts. A threshold nobody has backtested is a guess, and the first time you learn its volume is when it lands in the queue. ## What a per-entity baseline buys, and what it costs Per-entity means each account is compared with itself: alert when today's count exceeds that account's own p95, or exceeds its median by some multiple. It makes proportional change visible regardless of the account's absolute scale, which is exactly what a global number destroys. The costs are real and you should name them: - **State.** You must store and maintain a profile per entity, refresh it on a schedule, and decide how far back it looks. That is a data-management job, not a rule. - **Cardinality.** Tens of thousands of entities multiplied by several fields is a lot of keys, and the cost of computing the baseline grows with them. - **Cold start.** Every new account starts with no profile. The correct behaviour is to *withhold* the rule for that entity until it has a minimum history, not to alert on it. - **Small numbers.** A user whose median is one download a day trips a 4x ratio by opening four documents. Ratios over small counts are noise, so combine the ratio with an absolute floor: alert only when the count is both anomalous for the entity and above a level that could matter. - **Self-poisoning.** If the baseline is a rolling window and the adversary ramps slowly, their activity is inside the window and becomes the new normal. Guard this with a long-horizon cumulative check — total volume over ninety days against the same period last quarter — which a slow ramp cannot escape by being gradual. ## The shape that actually works In practice the deployable rule is a conjunction rather than a single test: an entity-relative departure, plus an absolute floor, plus at least one qualifying condition that raises the prior — an unfamiliar client, a session from an address the account has not used, activity outside the account's usual hours, or a departing employee flag. Each term is individually weak; together they produce a volume an analyst can actually work. And measure the rule by what it yields, not by whether it exists. If it has produced two hundred alerts and no escalations, the threshold is not tuned, whatever process produced it. ## The interview answer Say plainly that a global threshold is calibrated by the loudest legitimate user and therefore blind to proportional change in everyone else; that the fix is per-entity comparison derived from backtested history; and then, unprompted, name the costs — state, cardinality, cold start, noisy small numbers and a baseline the adversary can walk into. Candidates who only give the first half sound like they have read about baselining; candidates who give both halves sound like they have run one.