When is a deliberately frozen model the right choice over one that keeps updating itself?

## The question behind the question Every discussion of online learning assumes fresher is better. It usually is. But there is a class of systems where the right answer is a **fixed artifact**: parameters that do not change until a human decides they should, and then change through a reviewed release rather than by absorbing yesterday's rows. The deciding comparison is: *what does a version change cost, versus what does staleness cost?* Most teams only estimate the second. ## Why the cost of a version change can dominate **Revalidation is a committee, not a job.** A sepsis-alert model that nudges clinicians toward escalating care cannot change behaviour because a nightly update ran. Each version needs a clinical review of its performance across patient subgroups, a sign-off, and often a documented decision record. If that takes eight weeks of other people's time, the model that updates itself daily is proposing eight weeks of committee time per day, which simply means the committee stops being consulted. **Human calibration is part of the system.** Users learn what an alert means from experience: this system fires about this often, and when it does, roughly this fraction of the time it is right. Change the parameters and you change the alert rate and the precision at the operating threshold — the number itself may improve while the users' calibrated response silently becomes wrong. Frozen behaviour is a feature to the humans downstream. **Accountability needs a fixed object.** When a decision is challenged months later, someone must reproduce what the system did and why. A versioned artifact with a stored validation report answers that. A continuously updated parameter vector answers "it depended on the state of the learner at 03:14, which no longer exists." **Feedback loops.** This is the subtlest argument and the strongest. When the model's outputs influence the process that generates its future labels — an alert changes the treatment, a score changes who is approved and therefore whose outcome is ever observed — a learner that keeps consuming that stream is training on consequences of its own behaviour. It can confirm itself into a corner, and the effect is invisible in its own metrics, because the data agrees with it by construction. Freezing does not remove the loop, but it stops the loop from being closed automatically and gives a human the chance to notice. ## What freezing costs Be honest about the other side, or the answer sounds like risk aversion. A frozen model is not stable, it is *unchanging while the world changes*: the patient mix shifts, a lab changes assay, coding practices are revised, a new intake pathway appears. Performance decays without any visible event, and because nothing changed on your side, nobody goes looking. "Do nothing" is a decision with a risk profile, not the absence of one — and it is the decision whose risk is least likely to be measured. Freezing also atrophies capability. Teams that have not shipped a model version in eighteen months usually cannot: the training data has moved, the pipeline has bit-rotted, the person who built it has left. When evidence finally demands an update, the update takes a quarter instead of a week, which is exactly when you needed it to be fast. ## The middle ground worth arguing for The useful senior answer refuses the binary. Separate two things that online learning conflates: - **Learning continuously** — a learner may keep consuming data and producing candidate parameter snapshots. - **Serving a changing artifact** — what actually scores traffic changes only through a reviewed release. Run the learner in shadow: it never serves, its candidate snapshots are evaluated against a fixed reference set and the current artifact, and a human release decision promotes one. You keep the readiness and the evidence without letting the system reconfigure itself under the people who depend on it. Two supporting commitments make freezing defensible rather than negligent. First, freezing means *committing to measure*, not committing to ignore — a frozen model with no ongoing measurement is an unexamined liability. Second, keep the refit path exercised: rebuild the model periodically even if you do not ship it, so that the capability to update is proven and the update, when it comes, is a release rather than an excavation. ## How to answer this in an interview Name the axis (validation cost versus staleness cost), give the concrete conditions — regulatory or clinical sign-off, human calibration to a fixed behaviour, accountability for individual decisions, feedback loops through the label-generating process — then state the cost of freezing plainly, and land on the shadow-learner separation. The failure mode to avoid is treating "freeze" as caution and "update" as progress. Both are choices with a risk profile, and a principal is expected to price both.