Credit Along the Whole Trajectory

EWC computes importance at the finish line: train task A, freeze, estimate the Fisher at the final parameters, then protect. But a synapse's contribution to learning isn't determined at the endpoint — it's determined by everything that happened along the way. A parameter might move a lot early in training (doing real work reducing the loss) and then sit still at the minimum; an endpoint curvature estimate sees only the stillness. Synaptic Intelligence, from Zenke, Poole, and Ganguli, fixes this by keeping a ledger during training: each synapse accumulates credit for the loss decreases it participated in, integrated over the entire optimization trajectory.

The mechanism: a path integral per synapse

The core object is ωₖ, the importance of synapse k for the current task, defined as a path integral over the training trajectory:

ωₖ = ∫ gₖ(θ(t)) · θ'ₖ(t) dt

where gₖ is the gradient with respect to parameter k and θ'ₖ is that parameter's velocity. Intuitively: at each moment, multiply how much the parameter moved by how much the loss cared about that movement direction, and add it up. A synapse that consistently moved downhill accumulates large ωₖ; a synapse that jittered orthogonally to the gradient accumulates little. This is computed online, from quantities SGD already produces — no extra backward passes, no separate estimation phase.

At the end of each task, the raw ωₖ values are normalized into per-parameter regularization strengths:

Ωₖ^μ = Σ_{ν<μ} ωₖ^ν / ((Δₖ^ν)² + ξ)

The denominator (total squared movement of that parameter during task ν, plus a damping term ξ) puts the importance in the same units as the loss — it's essentially "loss decrease per unit of parameter movement," a scale-free measure of how much bang each unit of movement bought. Then training on the new task minimizes a surrogate loss:

L̃_μ = L_μ + c Σₖ Ωₖ^μ (θ̃ₖ − θₖ)²

where θ̃ₖ are reference weights (the parameters at the end of the previous task) and c trades off old versus new memories. After the update, ωₖ resets to zero and accumulation begins for the next task. The ω's update continuously; the Ω's and reference weights update only at task boundaries.

Why the trajectory beats the endpoint

The paper's supplementary analysis makes the comparison with EWC precise, and it's the kind of argument that sticks with you. For a quadratic loss, the diagonal of the empirical Fisher at the minimum is zero — at the bottom of a bowl, the gradient is zero, so gradient-outer-product importance sees nothing. But the path integral is proportional to the diagonal of the Hessian: it recovers the curvature, for free, from the trajectory. More generally, the Fisher is an endpoint estimate while SI integrates over everywhere the optimizer actually went. The authors show the two measures correlate in practice (explaining why both work), but SI's is the more faithful one — and it costs nothing beyond bookkeeping, since it reuses the gradients training already computes.

The biological framing is not window dressing. The paper draws on the idea that real synapses maintain local traces of their own involvement in learning — "intelligent synapses" with internal complexity beyond a single scalar weight — and that consolidation exploits these traces. Whether or not the neuroscience holds up in detail, it motivated the right algorithmic question: importance should be a local, online quantity, not a post-hoc global estimate.

Results and limits

On continual classification benchmarks (permuted and split MNIST variants), SI dramatically reduced forgetting relative to plain SGD while maintaining computational efficiency — the "no extra gradients" property means it scales like ordinary training, unlike EWC's Fisher computation phase or GEM's per-task backward passes. The paper reports SI performing comparably to EWC, consistent with the correlation between their importance measures.

The limitations are real and worth naming. The surrogate-loss derivation is exact only for two tasks; beyond that it's an approximation the authors validate empirically rather than theoretically. The strength parameter c has no principled value — c = 1 would be correct if the path integral were noiseless, but in practice it must be set smaller to compensate for estimation noise, which reintroduces the tuning problem EWC has with λ. The method still needs task boundaries to update Ω and θ̃, and it needs them to be reasonably clean. And like all regularization approaches, it fundamentally restricts plasticity to protect stability — it can never get positive backward transfer, where new learning improves old tasks, because the mechanism only ever says "don't move."

Importance isn't a property of a parameter at a point — it's a property of the path the parameter traveled. Integrate the gradient along the trajectory and you get curvature for free; estimate it at the endpoint and you get noise.

The arc

SI and EWC are the twin peaks of the regularization family, and their contrast defines the family's frontier: endpoint estimates versus trajectory integrals, extra computation versus free bookkeeping. Later work — Memory Aware Synapses (unsupervised importance), Riemannian Walk (combining both) — fills in the space between. But the deeper legacy is the framing: per-synapse credit accumulated during learning, from the learning signal itself. That's a short step from my own research question. SI asks "how much did this synapse contribute to reducing the loss so far?" The counterfactual version asks "how much would past performance degrade if this synapse changed, and how much would future performance gain?" — forward-looking, interventional, and trajectory-aware. SI built the ledger; the next step is to make it counterfactual.

Paper: Continual Learning Through Synaptic Intelligence — Zenke, Poole & Ganguli, 2017 (ICML).