How does elastic weight consolidation protect a network against catastrophic forgetting?

## The premise Rehearsal fights forgetting by putting the old data back into the objective. **Elastic weight consolidation (EWC)** fights it without the data, by adding a term that makes the *parameters* the old task relied on expensive to move. It is the canonical regularisation-based approach to continual learning and the one to reach for when the source examples cannot be kept. ## The objective During adaptation to a new task the loss becomes: `total_loss = new_task_loss + (lambda / 2) * sum_i F_i * (theta_i - theta_star_i)^2` where: - `theta_star` is the parameter vector at the end of training on the old task — a snapshot, - `F_i` is the i-th entry of the diagonal of the Fisher information matrix, evaluated at `theta_star` on old-task data, - `lambda` sets the overall strength of the constraint. Read it as a spring per parameter. Every weight is attached to where it was, and `F_i` is that spring's stiffness. Where stiffness is high the weight can barely move; where it is near zero the weight is effectively unconstrained and available for the new task. ## What the Fisher diagonal means and how it is estimated The Fisher information measures how sharply the model's log-likelihood responds to a change in a parameter. In practice the diagonal is estimated by running old-task inputs through the trained model and averaging the **squared gradient of the log-likelihood** with respect to each parameter: `F_i = average over samples of (d log p(y | x; theta_star) / d theta_i)^2` Two variants exist and it is worth being precise about which you mean. The true Fisher samples the label `y` from the model's own predictive distribution; the *empirical* Fisher uses the observed labels from the data. The empirical version is the cheaper and more common choice, and it coincides with the true Fisher when the model fits the old task well. The intuition: a large squared gradient means small perturbations of that weight change what the model believes about old-task data, so moving it is likely to cost old-task performance. A near-zero value means the old task is indifferent to that weight. EWC turns that indifference into permission. The estimation is a single pass over old-task examples, done **once**, at the moment the old task finishes. Afterwards you store `theta_star` and the Fisher diagonal — two vectors the size of the model — and the examples can be discarded. That is the whole storage argument for EWC, and it is also its honest caveat: EWC is not a no-data method, it is a *no-retention* method. You need access to old-task data at consolidation time. ## Why the penalty is not weight decay A frequent confusion. Weight decay pulls parameters toward **zero**, uniformly, and expresses a preference for small weights. EWC pulls parameters toward the **old solution**, with a per-parameter strength derived from how much the old task cared. They have the same quadratic form and completely different anchors, and swapping one for the other does nothing for retention. It is also not freezing. A frozen parameter cannot move at all; an EWC-constrained parameter can move if the new task's gradient is strong enough to pay the quadratic price. That graded, negotiable stiffness is the reason the method is called *elastic*, and it is what lets the network keep learning where the old task has no stake. ## Choosing lambda Lambda is the retention/plasticity dial and it has failure modes at both ends. - Too small: the penalty is negligible, the parameters drift, and forgetting proceeds much as it would without EWC. - Too large: the model is anchored to the old solution and cannot fit the new task — retention is perfect and the adaptation is pointless. There is no default that transfers between problems, because the scale depends on the loss magnitude and on the Fisher scale. Tune it against both evaluations: the new task's validation score and a retained source evaluation set, sweeping lambda and reading the frontier between the two. ## Where it falls short - **The diagonal approximation.** Only per-parameter importance is modelled; correlations between parameters are ignored. Real networks have strongly coupled weights, and a pair that must move together to preserve a feature is not represented by two independent springs. - **Accumulation across tasks.** With several past tasks you either keep a penalty per task, which grows the bookkeeping linearly, or fold them into a single running anchor. Either way the constrained fraction of the network grows over a long sequence and plasticity declines — the network gradually stiffens until new tasks stop landing. - **Task boundaries required.** The snapshot and the Fisher pass have to happen at a known moment. A continuously drifting stream with no clean boundary does not offer one. - **Weaker than replay.** When old data is available, interleaving it typically retains more for less tuning effort. EWC earns its place when the data cannot be kept, or as a complement that lets a smaller buffer go further. - **A local, quadratic picture.** The penalty is a second-order approximation of the old loss around `theta_star`. It is trustworthy near that point and progressively less so as the parameters travel far from it, which is exactly the regime a large new task creates. ## How to present it in an interview State the objective, say what the Fisher diagonal measures and how it is estimated, distinguish it from weight decay and from freezing, and name the diagonal approximation plus the accumulation problem as the limits. If you also say that you would still measure the result on a retained source evaluation rather than trusting the penalty, you have covered what the question is really probing: whether you understand that the method is an approximation with a knob, not a guarantee.