The Paper That Named the Problem

In 1989, two psychologists did something that should have been reassuring and turned out to be devastating. Michael McCloskey and Neal Cohen trained simple backpropagation networks on paired-associate lists — the same kind of A-B, A-C word-pair tasks experimental psychologists had used on humans for decades — and compared sequential learning against concurrent learning. When the network studied both lists interleaved, it learned everything. When it studied list A-B first and list A-C second, the second list didn't just degrade the first: it annihilated it. The network, prompted for a B response, confidently produced the C response instead. The old knowledge hadn't decayed. It had been overwritten.

This was the experiment that gave catastrophic forgetting its first name — the authors called it catastrophic interference — and nearly forty years later, it remains the cleanest demonstration of the core pathology.

What they actually did

McCloskey and Cohen ran two connectionist simulations, and the second is the famous one. It replicated a 1959 study by Barnes and Underwood on retroactive interference in humans. The setup:

• Task structure. The network learns A-B pairs (stimulus A → response B), then A-C pairs (same stimulus A → response C). A context pattern in the input vector distinguishes which list is being tested.
• Concurrent condition. A-B and A-C items interleaved during training. The network learned all associations correctly, no problem.
• Sequential condition. Train A-B to criterion first, then train A-C. After each A-C presentation, both lists were tested.

The result was not subtle. The amount of A-C training that brought human subjects to 50% correct brought the network's A-B recall to nearly 0%. And the errors weren't noise — they were systematic: the network gave C responses when asked for B. The A-C list had overwritten the A-B list in the same weights.

Then came the part that makes the paper genuinely scientific rather than merely alarming. The authors tried to fix it: hidden units, learning rate, overtraining on A-B, freezing weights, target values of 0.1/0.9 instead of 0/1. None of these manipulations satisfactorily reduced the interference. The failure was structural, not a hyperparameter accident.

Why the failure is structural

The paper's three conclusions still read like a diagnosis of modern fine-tuning:

• At least some interference will occur whenever new learning alters the weights involved in representing old learning.
• The greater the amount of new learning, the greater the disruption of old knowledge.
• Interference is catastrophic when learning is sequential but not when it is concurrent.

The first point is the one that matters most. In a distributed representation, knowledge of A-B and A-C lives in the same weights. Gradient descent on the new objective has no reason to respect the old function — it follows the new gradient wherever it leads, and the old solution is just a point it walks away from. Interleaving works because the gradient is then an average over both tasks; the weights settle where both are satisfied. Sequential training gives the optimizer amnesia by construction: the old task's gradient is simply absent.

This is worth sitting with, because it reframes what the last decade of continual learning has been doing. Every regularization method, every replay buffer, every architectural trick is a way of smuggling the old task's gradient — or a proxy for it — back into the room after the old data has left.

What's surprising in hindsight

Two things stand out reading the paper today. First, the authors were comparing against human memory, and humans don't do this. Humans show retroactive interference, but it's graded and partial — we don't forget the word "dog" because we learned "stool." The gap between human and network behavior was itself the finding.

Second, the manipulations that failed are instructive about what doesn't work. Adding hidden units didn't help — capacity alone doesn't buy stability, because the optimizer will still repurpose the new units for the new task and overwrite the old ones. Overtraining didn't help — a deeper minimum for A-B is still a minimum the A-C gradient walks away from. Freezing weights didn't help in their hands, though the idea (protect important parameters) would return, properly formalized, in EWC and Synaptic Intelligence nearly thirty years later. The failures map the space of dead ends so precisely that you can read the subsequent literature as a point-by-point rebuttal.

Catastrophic interference isn't a bug in backpropagation. It's what backpropagation does when the past stops providing gradients — the present optimizes, and the past is just parameters in the way.

Limitations and open questions

The paper is a diagnosis, not a cure, and it shows. The networks were tiny by modern standards, the tasks were toy paired-associates, and the authors offered no general mechanism for which weights matter for old tasks — that would wait for the Fisher-information and path-integral machinery of the 2010s. There's also a subtlety the paper doesn't fully resolve: is the right comparison really sequential vs. concurrent, or is it about the statistics of the data stream? Later work (notably McClelland, McNaughton, and O'Reilly's 1995 complementary learning systems theory) would argue the brain solves this with two systems — a fast hippocampal learner and a slow neocortical one, with replay interleaving the two. McCloskey and Cohen's concurrent condition is, in effect, what the hippocampus provides the neocortex during sleep.

One more open thread: the paper treats interference as purely destructive. But the same mechanism — new learning reshaping shared weights — is also what enables transfer. The stability-plasticity dilemma is a dilemma precisely because you can't have one without the other in a shared representation. GEM's later distinction between negative backward transfer (forgetting) and positive backward transfer (new learning improving old tasks) is the mature form of this observation.

Where it sits in the arc

Everything in continual learning is downstream of this chapter. Goodfellow et al.'s 2013 empirical study of forgetting in modern networks is a direct sequel. EWC cites it as the problem statement. The entire replay literature — from Robins' pseudorehearsal to modern episodic memory — is an attempt to recover the concurrent condition the paper showed was sufficient. And the recent LLM-era findings (that forgetting intensifies with scale, that instruction tuning alleviates it) are new empirical chapters in a story McCloskey and Cohen started: the story of what happens when a distributed learner meets the world one thing at a time, which is how the world actually arrives.

Paper: "Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem" — McCloskey & Cohen, 1989 (in G. H. Bower (Ed.), The Psychology of Learning and Motivation, Vol. 24, pp. 109–165). No arXiv version exists; the chapter predates the archive by two years.