Rolling the dropout early-warning model back to its previous version mid-incident: what must that rollback cover beyond the trained artifact?

## What "the model version" actually names Ask three engineers what they would roll back and you get three answers: the code revision, the trained artifact, the feature-definition version, or the version stamped on decisions already made. In a serving tier, the thing that produces a prediction is none of these alone. It is a **pin set** - a small bundle of things that were evaluated together and are only known to be correct together: - the **trained artifact** (weights or tree structure, plus whatever the loader needs); - the **preprocessing** that turns a raw request into the model's input, which usually ships as code; - the **feature-definition version** the online path reads, which decides what `attendance_rate` means this month; - the **decision threshold** that turns a score into a flag, tuned per version against a target caseload; - the **output contract** the downstream consumer parses. A rollback that restores only the first item produces a combination that was never evaluated: last quarter's artifact scoring this quarter's feature definitions through this quarter's threshold. That is a new, untested configuration created in the middle of an incident. ## Rolling back a pointer, not rebuilding a build The fast rollback is a **pointer flip**: the serving configuration names an active pin set, you set it to the previous one, and replicas pick it up. It is not a retrain, and it should not be a build. Anything that requires compiling, re-fitting or re-deriving features puts a pipeline between you and the mitigation, and pipelines fail at exactly the wrong moment. The common trap is that code and model roll back independently: | What moved | Rolled back by | What is left behind if you forget | |---|---|---| | Service code | Redeploying the previous release | The pointer still resolves to the newer model | | Model artifact | Flipping the version pointer | New preprocessing still runs on old weights | | Feature definition | Restoring the prior definition version | Online values silently change meaning | | Decision threshold | Restoring it with the pin set | Caseload volume jumps even on a correct model | Because of the first row, "we redeployed the last good release and the predictions did not change" is a routine and confusing incident moment. It is the expected outcome whenever the model is fetched at runtime by a pointer. ## Preconditions that must already exist A rollback is a capability you build before the incident, not a command you invent during one: 1. **The prior version is retained and loadable.** Retention policies that age out artifacts, and loaders that only accept the current serialisation format, both turn a rollback into a retrain. 2. **Its feature definitions are still computed.** If the newer version replaced a definition rather than adding one beside it, the older model's inputs stopped existing. Adding definitions rather than mutating them is what keeps the previous version runnable. 3. **The pin set is stored as one unit.** If the threshold lives in a config file, the features in a platform, and the artifact in a store, nothing guarantees they move together. 4. **The flip needs no build.** One configuration change, effective on the next request or the next scheduled run, with a known propagation time you have measured. 5. **Someone reachable can do it.** A rollback only the model's owning team can perform is unavailable at the hours incidents favour. ## When a version rollback is the wrong lever Rolling back reverses a change **you** made. It does nothing when the cause is upstream and shared: a calendar feed that changed meaning, a schema that shifted, a source system that stopped emitting an event. Every retained version reads that same broken input, so every one of them is wrong in the same way, and the pointer flip buys nothing but a false sense of mitigation. The tell is simple and worth checking before you flip: *did anything about this model change, or did the world under it change?* If the last model change predates the suspected start, the lever you want is not the prior version - it is switching off the model path entirely and serving from something that does not read the broken input. The second limit is scope. A rollback restores prediction quality going forward. It does not touch decisions already delivered, and it does not re-score the weeks that went out wrong. Those are a separate, deliberate remediation step, and skipping it is how an incident gets closed while its effects are still working their way through an advising team.