MÖzel.
Does local learning
help models forget less?
AI models tend to forget old skills when they learn new ones. I taught a small network handwritten digits in five rounds (0 or 1, then 2 or 3, and so on) and checked how much of the early rounds it lost. I compared backpropagation, the standard way to train, with a brain-inspired “local” rule, where each connection learns only from its neighbours.
A reworked version of the local rule forgot a little less: about 45 points of accuracy lost on early rounds, against 51. But mostly because it changed the network more gently, like writing lightly over old notes instead of pressing hard. When I made a backprop-like version take smaller steps too, it did just as well. So most of the gain came from gentler learning. Whether being local helps at all is still an open question.
How the answer changed along the way
- Planned test: no difference. My first local rule turned out to do almost the same maths as backpropagation.
- A truly different rule forgot about 5 points less, but also moved the network’s connections about four times less.
- Smaller steps explained most of that gap. A small part is still unexplained, and only one of three test setups could measure it.
