Catastrophic Forgetting

## The phenomenon A network is trained on task A until it performs well. Training then continues on task B — new labels, new data, or simply a newer slice of the same stream. Performance on B rises as expected. Performance on A, which nobody is measuring because A's data is no longer in the loop, falls hard. That collapse is **catastrophic forgetting**: the loss of previously acquired capability caused by learning something new in the same set of weights. The word *catastrophic* is doing real work. Forgetting in a neural network is not the slow decay a person experiences; it can take a network from 0.91 recall to below 0.5 on the old task inside a single epoch of the new one. ## Why it happens Think about what the optimizer is actually asked to do. During fine-tuning the objective is the average loss over the **new** task's batches. There is no term in that objective that mentions the old data, so there is no force at all resisting movement away from the old solution. The old solution was one point in a large region of weight space where task A's loss is low; the new gradient points wherever task B's loss falls fastest, and that direction is essentially unrelated to A's low-loss region. Three structural facts make it worse: - **The parameters are shared.** A single backbone encodes features for both tasks. There is no partition of weights reserved for A, so any update that helps B may overwrite a feature A depended on. - **The objective is myopic.** Stochastic gradient descent optimises the current batch's loss. It has no memory of the loss surface it came from, and no notion that some directions are cheap for B but expensive for A. - **The output layer usually changes.** Adapting to a new label set typically means a fresh, randomly initialised head. Early in training that head produces large, badly aimed gradients that propagate into the backbone, and if the old head was discarded the old task's predictions are gone by construction, not merely degraded. ## Two distinguishable failures It is worth separating **representation drift** from **head drift**, because they are diagnosed and repaired differently. - *Head drift*: the backbone still encodes everything the old task needed, but the classifier on top has been re-aimed at the new labels. In a class-incremental setup, where each stage introduces new classes and only those classes appear as positives, the final layer picks up a strong recency bias — logits for recently seen classes systematically outrank older ones. - *Representation drift*: the features themselves have moved, so the information the old task needed is no longer linearly available. A cheap way to tell them apart is to freeze the current backbone and fit a fresh linear classifier on old-task data. If that probe recovers most of the old performance, the features survived and the head was the casualty. If it does not, the representation itself has been overwritten. ## Where the severity comes from Forgetting is not a constant; it scales with how far training is allowed to move the weights and how far apart the tasks are. - **Distance between tasks.** Two tasks that need similar features interfere less. A new suction-gripper skill that reuses the same visual features as the old parallel-jaw skill damages less than one that needs an entirely different notion of graspability. - **How much of the network moves.** The more layers are unfrozen and the larger the steps, the more the pretrained features can be displaced. Lower layers, which encode generic structure, typically drift less than the task-specific upper layers, partly because their gradients are smaller and partly because generic features remain useful for the new task. - **How long training runs.** Forgetting compounds with steps taken on the new distribution, so a long fine-tune on a small new dataset is a common way to destroy a good backbone. - **Sequence length.** Under repeated sequential adaptation — five stages, then ten — the earliest stage is usually the worst hit, because it has had the most subsequent updates to survive. ## What it is not Two confusions are worth naming. First, forgetting is **not** overfitting. An overfit model has memorised the training set of the task it is being trained on; a model that has forgotten may generalise perfectly well on the new task while having lost a different one. They can occur together, but neither implies the other. Second, forgetting is **not** literal erasure. The weights still carry a great deal of old-task structure, which is why old performance is often partially recoverable from a small amount of old data far faster than it was originally learned. That partial recoverability is exactly what the standard mitigations exploit: keeping a slice of old data in the training stream, or adding an explicit penalty that makes the parameters the old task relied on expensive to move. ## The practical consequence Because the new-task metrics look healthy the whole time, forgetting is invisible unless you deliberately keep measuring the old task. Any pipeline that adapts a model repeatedly needs a retained evaluation set from the original task, scored on the same cadence as the new-task validation set, or the regression ships.