ML Problem Framing

One row must fix three things: the prediction unit (one account in one billing month), the horizon the label covers (cancels within the next 30 days), and the as-of timestamp that separates feature territory from label territory.

Reserve first what you cannot spend — the caller's network round trip, fixed request handling and explicit headroom — then allocate the remainder across feature lookup, inference and post-processing. The inference slice becomes a ceiling any candidate model must fit.

The offline sweep metric ranks candidate models during training - a ranking score over a frozen retrospective cohort. The online launch metric decides whether the service ships: what the deployed worklist changed in care, read on live shifts.

Shipping the rule first delivers the product immediately and starts the clock on the evidence a model needs: every promise logged with the inputs as they stood at decision time, joined later to the actual delivery. Without that log, a model has no honest training set and no measured bar.

The arrival rate and the burst window the percentile must hold at, not just the percentile. A latency number without a load attached is met at the shift average and missed at the minute the business actually cares about.

Three divergences do it: the offline score integrates over a ranking the ward never reads past row twenty, the retrospective cohort is not the population the live path scores, and changed care counts action taken, not ranking accuracy.

Because the feature records a consequence of the cancellation, not a cause. Offline it separates the classes almost perfectly; at prediction time the account has not cancelled yet, so the field holds a neutral value and the model scores confidently on nothing.

A p99 budget is measured end to end at the caller, so it also covers the handheld's network round trip, request handling, feature lookup, post-processing and serialisation. Inference is one line item inside that total, not the total itself.

Forty parcels a week is too little evidence to key an estimate on: transit days are whole numbers, so a small shift in the week's mix flips which day sits in the middle and the quoted median hops. A support floor with a backoff to a coarser key fixes it.

Because the signals age at wildly different rates: slot inventory is wrong within seconds, aisle congestion within tens of seconds, item velocity within a day, bin geometry within a month. One shared number either breaks the fastest signal or over-builds for the slowest.

There is no single event: an explicit cancellation is logged with a timestamp, but a silent non-renewal is an absence. The label must be a predicate over billing state, declared positive once the renewal date plus a grace period has passed with no new paid period.

Every row whose as-of timestamp is newer than 44 days ago. The horizon plus the grace period is the label maturity lag, so the training cut sits 44 days in the past and the most recent six weeks of data carries features but no trustworthy outcome.

Neither reading is wrong; they answer different questions. The reversal comes from the candidate moving worklist rows toward the higher-prevalence ward, so the honest read is whichever matches how worklist capacity is actually allocated - and both get reported.

Decline when the reachable margin is small against the upkeep a learned estimator adds permanently: a maintained training set, a refit rota, monitoring, and someone on call to explain a bad date. State the margin that would change the answer before the offline sweep, not after.

As much weight as its track record earns. Pair each past launch's offline delta with the online delta that followed; that history says whether the proxy can license a launch or only veto a candidate, and what a launch carried by it must commit to.

A hub consolidation is a structural break: most of the model's training rows describe routings that no longer happen, while an eight-week rolling median only remembers eight weeks and re-converges as post-change parcels fill its window. Short memory wins across a break.

From the value side: what one prediction changes, net of the rule already in place. For pick-path routing that is picker seconds saved, times the fraction of picks whose order the model actually alters, valued at the loaded labour rate.

Team-draft interleaving needs many independent impressions and an action attributable to one ranker's item. A ward worklist is one shared list per team per shift, worked top to bottom, and each action changes the patient - so neither condition holds.

It gives up the guarantee that the target is the business outcome. The proxy matures in weeks and fires while an account is still winnable, but it flags seasonal dips, misses steady users who leave over price, and moves whenever the product changes.