What must a load forecaster's night runbook pre-decide so an on-call engineer can roll back the model version alone?

## Why the decision is made in advance The responder at 03:00 has one page, no colleagues, and no way to tell whether the model degraded or the data feeding it did. Diagnosis at that hour is slow and unreliable, and every interval spent on it is published at whatever quality caused the page. A night runbook is therefore not documentation of how the system works - it is a pre-made decision that is acceptable under every reading of the evidence the responder can actually gather. That means the runbook is written when people are rested and have the counterfactuals in front of them, and reviewed whenever the model, the fallback or the decision clock changes. ## Which fallback, and why the default matters | target | cures a bad promotion | cures a dead or frozen feature feed | accuracy cost on a normal night | time to take effect | |---|---|---|---|---| | previous model version | yes | no - it reads the same feature path | small | next interval | | seasonal-naive baseline | yes | yes - it reads only past actuals | significant | next interval | | do nothing, keep serving | no | no | none, but the error continues | - | The asymmetry is the whole point. A version rollback assumes the fault was introduced by the promotion; if the real cause is upstream, the previous version consumes the same poisoned inputs and fails the same way, and the responder has spent the night's one action for nothing. The non-learned baseline is immune to the feature path, which makes it the safer default under uncertainty even though it is less accurate when everything is healthy. The runbook should state the default explicitly, and state the evidence that would justify the other choice - typically a promotion within the last few days with no accompanying input-side signal. ## What a rollback has to cover A model version is rarely just weights. Rolling back partially is a common and expensive mistake, so the runbook enumerates the set: 1. **The model artifact** actually loaded by the serving path. 2. **The feature definitions and transformation code** the candidate introduced - a rolled-back artifact reading newly-shaped features can be worse than either version on its own. 3. **Any post-processing** that shipped with the promotion: clamps, calibration adjustments, rounding to market granularity. 4. **The alarm's own thresholds**, if they were retuned to the new version's error profile, or the restored producer will look broken against a rule written for its successor. 5. **The provenance flag** on every published forecast, so downstream consumers and the morning review can tell which producer served which interval. ## The authority line The expensive failure is not a wrong rollback; it is an hour spent looking for someone to approve a reversible action. The runbook should therefore say, in plain terms, that the named on-call may switch producers without approval, within a declared blast radius - which horizons, which regions, which intervals. Reversibility is what earns that: if the switch takes effect next interval and can be undone next interval, the cost of being wrong is bounded and small, while the cost of hesitating accrues every interval. The converse also belongs in the runbook. Where an action is irreversible - anything that touches an already-submitted market position - a second person is justified, and the runbook names who and how to reach them rather than leaving the responder to invent an escalation path at 03:00. ## What the night forbids Just as important as the permitted action is the list of things the responder must not attempt: - **retraining or refitting anything** - the refresh policy is a standing mechanism with its own gates, not a night-time improvisation; - **editing the alarm's thresholds to clear the page** - that destroys the record of what fired and why, and the change escapes the review a promotion would get; - **rolling back one component and not the rest**, producing a configuration nobody has ever tested; - **deep root-cause analysis** before mitigating - the forecast is being published continuously, and diagnosis is a morning activity once the bleeding has stopped. ## Making the fallback temporary by construction A fallback that outlives the night becomes the system by inertia, and the operator ends up running a seasonal-naive forecaster for a month without deciding to. So the switch carries an **expiry** and a scheduled morning review that must either restore the model, extend the fallback deliberately, or open a defect. The fallback's own quality sits on the same alarm, which means the degraded accuracy stays visible instead of becoming the new normal against which everything is judged.