Immunity by Construction

The most honest solution to catastrophic forgetting is also the most brute-force: if learning a new task destroys the old weights, don't touch the old weights. Give the new task its own network, freeze everything that came before, and wire the old networks' features into the new one as optional inputs. That is the entire idea of progressive neural networks, and its honesty is its virtue — it achieves immunity to forgetting not by clever regularization but by construction, and in doing so it clarifies exactly what the clever methods are trying to approximate.

The architecture

A progressive network grows one column per task. Column 1 is an ordinary deep network trained to convergence on task 1, with parameters Θ⁽¹⁾. When task 2 arrives, Θ⁽¹⁾ is frozen — made constant as far as the optimizer is concerned — and a fresh column with random initialization is instantiated. Each layer of the new column receives two inputs: its own previous layer, and the previous layers of all earlier columns, via lateral connections:

hᵢ⁽ᵏ⁾ = f(Wᵢ⁽ᵏ⁾hᵢ₋₁⁽ᵏ⁾ + Σⱼ₍ₖ Uᵢ⁽ᵏⱽʲ⁾hᵢ₋₁⁽ʲ⁾)

Because lateral connections only run forward in column-index (from j < k to k), and because earlier columns' parameters never receive gradients, there is no interference between tasks. Period. No forgetting is possible, because the thing that forgets — gradient updates to shared weights — has been removed from the picture.

A few engineering details matter. Raw lateral connections across columns with different activation scales condition poorly, so the authors add adapters: each lateral input is scaled by a learned scalar initialized small, then passed through a single-hidden-layer MLP (1×1 convolutions for conv layers) that also performs dimensionality reduction. This keeps the lateral parameter count on the order of one column's parameters rather than exploding quadratically. The new column is free to reuse, modify, or ignore old features — reuse is encouraged by the wiring, never forced.

Why it beats finetuning at its own game

The paper's real target isn't forgetting per se; it's the pretrain-and-finetune paradigm. Finetuning is destructive: it discards the previously learned function, and it bakes in an assumption that the tasks overlap enough for the old initialization to be a good start. Progressive nets make no such assumption. Tasks can be orthogonal or even adversarial; the new column simply learns to ignore the lateral inputs it doesn't need.

The experiments — extensive, in deep RL on Atari and 3D maze games — show progressive networks matching or slightly beating finetuning on transfer without the destructive consequences. And because old features are never destroyed, the authors could do something finetuning makes impossible: study where transfer actually happens. Using Average Fisher Sensitivity (AFS) — the diagonal Fisher of the policy with respect to normalized activations, measuring how sensitive the policy is to each layer's representation — plus a perturbation analysis, they showed transfer occurring at both low-level sensory layers and high-level control layers of the policy. That's a genuinely new empirical observation about deep RL transfer, enabled entirely by the architecture's non-destructiveness.

The price of immunity

The paper is admirably upfront about the cost, in a section literally titled "Limitations":

• Parameters grow with tasks. Each task adds a column. The authors' own analysis shows only a fraction of new capacity is actually utilized, and the fraction shrinks as columns accumulate — suggesting pruning, smaller columns, or online compression as remedies, all left as future work.
• Inference needs the task label. The network can solve all K tasks at test time, but you have to know which column to query. Task-agnostic inference — the setting where the model must also figure out which task it's doing — is not addressed.

There's a deeper limitation the paper doesn't dwell on: progressive nets achieve stability by abandoning plasticity in the shared sense. Old columns never improve from new experience — there is no positive backward transfer, no consolidation, no shared representation that gets richer. It's a growing ensemble, not a single mind that accumulates. For a handful of Atari games that's fine. For a lifetime of tasks, the linear growth is a dealbreaker, and the field largely moved toward methods that keep one set of weights and protect them selectively.

Progressive networks are the control experiment for the whole field: total stability, zero forgetting, linear cost. Every regularization method since is an attempt to get the stability without paying the linear cost — to be progressive without growing.

Why it still matters

Three reasons. First, it's the cleanest existence proof that transfer and immunity can coexist — you don't have to choose between leveraging the past and preserving it. Second, the lateral-connection pattern became a design primitive: PathNet, adapters in NLP (Houlsby adapters are spiritual descendants), and modern modular approaches all inherit the idea of frozen substrate plus trainable interface. Third, and most relevant to today's LLM era, the paper anticipates the current architectural moment. Freezing a pretrained backbone and training small added components — LoRA, adapters, and most recently memory layers and width-upscaling schemes like SCALE — is progressive networks' core move, rediscovered at scale. The difference is that modern variants share the frozen substrate across tasks rather than growing a column per task, which is exactly the compression the authors' appendix suggested.

For my own work on credit assignment in lifelong networks, the paper poses the sharpest version of the question: if you could afford the columns, you'd never need to decide which weights matter. The entire regularization literature — EWC's Fisher, SI's path integral, MAS, all of it — exists because we can't afford the columns, and so we must solve the harder problem of apportioning credit and protection within a fixed capacity. Progressive nets show what the answer looks like when capacity is free.

Paper: Progressive Neural Networks — Rusu, Rabinowitz, Desjardins, Soyer, Kirkpatrick, Kavukcuoglu, Pascanu & Hadsell, 2016.