You removed a partner's poisoned records from the training table, but the next retrain is two weeks out — what happens in between?

## The gap nobody writes down There are two clocks in a train-time remediation and they run at completely different speeds. A control on the request path is live in minutes. A correction to the training data is live *at the next training run*, and only if that run consumes the corrected snapshot. Between the moment you delete the rows and the moment a retrained model is serving, the deployed weights are exactly what they were: fitted to the contaminated table, answering ordinary requests with the skew the adversary paid for. On a same-day grocery forecaster with a fortnightly retrain, that window is two weeks of staffing decisions taken on numbers you already know are wrong for some regions. The dangerous part is not the exposure itself — it is that the ticket says "contaminated records removed" and everybody reads that as *done*. ## The four things you can actually do **Roll back to an earlier checkpoint.** The fastest real fix, because an earlier checkpoint was fitted before the rows existed. Its precondition is a date: you must know when the partner's records first entered a training snapshot. Without that, rollback is a guess, and landing on a checkpoint trained on the same rows changes nothing while creating the appearance of action. The cost is everything legitimately learned since that point — on a demand forecaster, weeks of real seasonal and promotional signal, which is itself a forecast-quality hit. **Retrain off-cycle.** Correct in principle and often the right call, but it is not free: compute, a revalidation pass, and the risk of shipping a model that has had far less scrutiny than a scheduled release. It is also only as good as the audit — if you removed the rows you *found* and the same channel supplied others, the off-cycle run relearns them and you have spent the cycle for nothing. **Scope a fallback.** If the skew is confined to a slice — one partner's regions, one product family — you can route just that slice to a simpler baseline (last week's demand, a seasonal average) and leave the model serving everywhere else. This is usually the cheapest honest answer, because a global fallback degrades forecasts for the whole estate to fix a localised problem. **Bound the downstream effect.** The consumer is a staffing service, and a compensating limit — no more than an X percent move from the previous week's booking without a human look — caps what a wrong forecast can actually cost, without touching the model at all. Note what this is: a control on the *consumer*, not on the model or the request path. It is a legitimate stopgap and it should be recorded as one. ## The precondition all four share Suspend or quarantine the write channel first. If the partner keeps filing records into the training table, the audited snapshot is stale the moment you take it, the off-cycle retrain relearns the effect, and the rollback checkpoint is superseded by the next scheduled run. The order matters: stop the writes, then correct the data, then fix the model. ## What is different about the adversary during this window They do not have to do anything. This is the property that distinguishes a train-time compromise operationally: there is no ongoing activity to detect, no traffic to throttle, no session to kill. Watching the request logs during the serving window will show you nothing, and that absence is the expected observation rather than a reassuring one. Anyone proposing to "monitor closely for two weeks" is proposing to watch the wrong channel. ## Dating the contamination is the hard part Almost every option above turns on one question: when did those rows first appear in a snapshot the model was trained on? Answering it requires knowing which snapshot each deployed checkpoint consumed. Where that lineage exists, rollback becomes a five-minute decision. Where it does not, you are choosing between retraining blind and rolling back further than you need to — and the cost of not having the lineage is paid entirely inside the incident, which is worth saying out loud afterwards. ## Verifying the fix, not assuming it When the retrain lands, aggregate error holding flat is not evidence the effect is gone. A localised skew barely moves an aggregate, so flat aggregate error is exactly what you would see whether or not it survived. The check has to be on the affected slice, against a reference from before the contamination — and if you cannot define the affected slice, you cannot verify the fix, which is a reason to establish it early rather than at the end. ## How to write it up The remediation section should carry three things a request-path fix never needs: the date the deployed model becomes clean, what is serving in the meantime, and who accepted the window. That is the honest shape of a train-time remediation, and it looks nothing like the one-line rule deployment that closes an inference-time finding.