AI & Machine Learning · Hopfield Networks: How a Machine Remembers
Scramble a face, let 256 simple switches follow one rule, and watch the face come back — no address, no search, nobody in charge.
Prologue
A computer finds a thing by knowing its address. Row 4, shelf 2, box 19. If you forget the address, the thing is gone forever, even though it is sitting right there.
You do not work like that. Someone hums four notes and a whole song arrives; you glimpse one corner of a face in a crowd and the whole person snaps into place. You never knew an address — you had a piece, and the piece pulled the rest in after it.
In January 1982 a physicist named John Hopfield sent a five-page paper to the Proceedings of the National Academy of Sciences. He was not an AI researcher: he had worked on solid-state physics (he coined the word polariton) and on the chemistry of haemoglobin. His question was stranger: what can a large pile of very simple parts do together that none of them can do alone?
That pile of simple parts is what you are about to become: 256 switches on this page. Each switch feels only its wires, has never heard of a picture, and follows exactly one rule.
Out of that — with nobody in charge, no address, no search — a face comes back.
Act one
Here is your entire job. You have a wire to every other neuron, and each wire has a strength: positive (“when you're on, I want to be on”) or negative (“when you're on, I want to be off”). Add up what your neighbours are shouting, weighted by those strengths, and you get your local field — the total push on you.
If the total is positive, you switch ON. If it is negative, you switch OFF. That is the whole of you.
You cannot see the picture, and neither can any other neuron, so it is fogged out below. Take six turns anyway.
Real arithmetic on a real 256-neuron network. Your neuron is picked at random each turn.
What neuron {id} can see
The picture you are inside (you cannot see this)
256 switches, 16 by 16. Warm = ON, dark = OFF. The fog lifts after your sixth turn.
Here is the uncomfortable part: whatever you chose, the network almost certainly repaired that pixel behind your back on a later turn. Hopfield wrote that the model is “only weakly sensitive to details of the modeling or the failure of individual devices.” One neuron does not matter — and that is the point.
Act two
So where does the picture live? Not in any neuron. It lives in the wires — and every memory lives in all of the wires at once, laid on top of every other memory like transparencies on an overhead projector.
The rule for laying one down is simple, and it is close to what the psychologist Donald Hebb proposed for real brains in 1949 (later summed up as “neurons that fire together, wire together”). For every pair of neurons, look at the picture you want to keep. If the two pixels agree — both on or both off — add a little to the wire between them; if they disagree, subtract a little.
Hopfield's storage rule from the 1982 paper. His neurons were 0 or 1, and (2V−1) turns that into −1 or +1; this page uses −1 and +1 directly, which is the same network. Sum over every memory s; a neuron never wires to itself.
Notice what this rule does not do: it never checks for collisions, never allocates space, and never counts how many memories are already in there. It just adds, so every memory is smeared across all 65,280 wire strengths and every wire carries a trace of every memory. Pull a wire out and you don't lose a memory — you lose a tiny sliver of all of them.
Click pictures to store or remove them. The web on the right is the real weight matrix, live.
The memories on the table
The wires (a 256 × 256 weight matrix)
Each dot is one wire between two neurons: warm means “agree with me”, cool means “disagree with me”. Watch a whole memory dissolve into texture the moment you add it.
Act three
Networks like this had been built before — by Karl Steinbuch in 1961, by David Willshaw and colleagues in Nature in 1969, by Shun-ichi Amari, Kaoru Nakano, Teuvo Kohonen and James Anderson all in 1972, by William Little in 1974. The wiring rule was not new. What Hopfield brought was a physicist's habit, and it changed everything.
He insisted the wires be symmetric: whatever neuron 43 says to neuron 8, neuron 8 says back to 43. That one assumption is not especially biological — real synapses aren't like that — but it lets you write down a single number for the whole network, exactly the way physicists write down the energy of a magnet:
Now the trick. Take any neuron and let it follow the rule: because the wires are symmetric, that move can never raise E. Every flip either lowers the total or leaves it exactly where it was.
So the network cannot wander forever or loop: it is a ball on a hillside, and every update is a small roll downhill. It must stop at the bottom of some valley. And Hebb's rule, it turns out, digs a valley at every picture you stored.
Read that again with the ball in mind. “Remembering” is not a search. It is a fall.
One neuron updates at a time, in random order, exactly as in the 1982 paper.
The network right now
Energy, falling
How much it looks like each stored memory
The energy line only ever goes down or flat — never up. If you ever see it rise, something is broken. It won't.
Act three, continued
One detail in the 1982 paper looks like housekeeping but holds everything up: neurons update one at a time, in random order. If they all jump at once, the downhill guarantee can fail. Make a prediction below, then test it.
Two switches joined by one wire that says “agree with each other”. They start out disagreeing.
Prediction: if both coins jump at the same moment, will they ever settle?
Act four
When the Nobel Prize committee explained this in 2024, they used Hopfield's own picture: “rolling a ball through a landscape of peaks and valleys, with friction that slows its movement.” Here it is more than a metaphor. With two memories stored, the energy depends on just two numbers — how much the state resembles memory A and how much it resembles memory B — so we can draw the whole landscape exactly.
The floor below is that formula: every point is a real state of a real 256-neuron network, and the height is its real energy. The four deep corners are memory A, memory B and — for reasons you'll meet soon — their two photographic negatives. Drop the ball anywhere and watch it choose.
Drag to spin the view. Click anywhere on the floor to drop the network there.
The network, actually
Slide the two memories toward each other and watch the floor stretch. When memories resemble one another, the valleys lean together and merge — which is the whole reason a Hopfield net finds similar things hard.
Act five
Every new memory digs a new valley — but it also throws dirt into all the old ones. Hopfield ran this on a machine with 30 and 100 neurons and reported the number that everyone still quotes:
Three years later Daniel Amit, Hanoch Gutfreund and Haim Sompolinsky did the full statistical-mechanics calculation and pinned it down: the critical load is αc ≈ 0.138. Past 13.8 memories per 100 neurons, recall doesn't get gradually worse. It falls off a cliff — a genuine phase transition, like ice becoming water at a temperature, not a gentle fading.
The bench below is not a picture of that result. It measures it, in your browser, right now: it builds fresh networks of 256 neurons, loads them with random patterns, damages each one, lets it settle, and counts.
Every point is a freshly built and freshly tested network. Nothing here is pre-baked.
Act six
Here is a fact that falls straight out of the arithmetic and delights everyone who meets it. If a picture is a valley, then its exact photographic negative is also a valley — every switch flipped, precisely as stable, precisely as deep. Hopfield noticed it immediately and waved it off in a parenthesis:
Teach it a face and you have silently taught it the anti-face too. Stranger still are the mixtures: take three stored memories, let the majority win at each pixel, and the network happily rests there — a confident state that is a blur of three things and a picture of none. Amit, Gutfreund and Sompolinsky showed in 1985 that these blends are built into the model.
Nobody put them in. They are the potholes the valleys make where they meet. Below, we drop a network into 240 completely random starting states and see where each one comes to rest.
100 neurons, random memories, 240 random starting states, settled to the bottom.
Meet a ghost
A
B
C
Act seven
On 14 July 1983, Nature printed two papers on one idea. In the first, Francis Crick — of DNA fame — and Graeme Mitchison proposed that dream sleep's job is not to remember but to forget: the sleeping brain fires more or less at random, finds junk states, and weakens them. Their phrase for it: “we dream in order to forget.”
Forty-seven pages later in the same issue, Hopfield, David Feinstein and Richard Palmer tested the idea on exactly the kind of network you now know. Start from a random state, let it fall wherever it falls, then apply Hebb's rule backwards to wherever it landed, weakening every wire that agreed. Repeat a few hundred times.
Random starts land in ghosts far more often than in real memories, so unlearning eats the ghosts first and barely touches what you wanted. The memories don't get stronger; everything else gets weaker, which looks the same from underneath. In their test, how easy the memories were to reach first differed by a factor of 3, and after unlearning by only 1.4.
And if you keep going? Try it. There is a cliff at the far end of the slider, and it is worth finding.
The same 100-neuron overloaded network, before and after a night's sleep.
Each bar is 240 fresh random starting states, settled all the way down, and sorted by where they stopped.
Speed round
You know the rules now, so call the landing before it happens. The network below holds four memories — face, key, tree and bird — and each round it gets a damaged picture. You have 8 seconds to predict where it will settle: one of the four, a negative, or a ghost that is none of them.
Nothing is scripted: after you choose, the network really settles, and the answer is wherever it lands.
Six honest wobbles
A beautiful wrong mental model is worse than an ugly right one. Here is where to distrust what you have just played with.
αc = 0.138 assumes memories that are random and unrelated, but our six icons share a mostly empty background, so they resemble each other far more. Four fit comfortably in 256 neurons; the fifth already starts to break things, nowhere near the 35 that 0.138 × 256 promises. Real memories are correlated, and that makes them harder to store.
The guarantee that energy always falls depends on Tij = Tji, but real synapses are one-way and not matched in pairs. Hopfield knew this, and his simulations showed the network still mostly works when the symmetry is broken. The tidy “it must come to rest” proof, though, is gone.
Steinbuch (1961), Willshaw and colleagues (1969), Amari, Nakano, Kohonen and Anderson (all 1972) and Little (1974) built similar machines first. What 1982 added was the energy function and a short, vivid paper that drew a generation of physicists into the field.
Hopfield's 0.15 was measured in simulations that allowed a few wrong bits; the 0.138 of Amit, Gutfreund and Sompolinsky is an exact result for very large networks. Demand every pattern back perfectly and McEliece and colleagues showed in 1987 that capacity drops to about N ⁄ (4 ln N) — roughly eleven for 256 neurons.
Crick and Mitchison's idea about dream sleep was a hypothesis, and it has never been established; Hopfield, Feinstein and Palmer showed only that the mathematics could work, not that brains do it. More than forty years of sleep research have not settled the question.
Nothing in a 2026 language model is a classic Hopfield network; the family line runs sideways, through Boltzmann machines (1985) and the dense memories of Krotov and Hopfield (2016), whose capacity grows far faster than N. In 2020 Ramsauer and colleagues showed that a continuous modern version updates with the same formula as a transformer's attention step — a family resemblance, not direct descent.
The long thread
The examination
One last thing
On 8 October 2024 the Royal Swedish Academy of Sciences gave the Nobel Prize in Physics to John Hopfield, then 91, and Geoffrey Hinton, “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” A physics prize, for a memory made of switches.
You have now been every part of that memory: the switch that couldn't see, the wire that held a smudge of everything, the ball and the hillside. You watched a face come back out of noise with nobody searching for it. And you watched the same machine invent a face that was never there, using exactly the same rule.
So here is the question. When you recognise someone across a station platform in the wrong coat, in bad light, after eleven years — is anything in your head searching? Or are you also just falling into a valley you dug a long time ago, and calling the landing “remembering”?
Every date, number and quote on this page was checked against these sources.