You are standing in the crowd
The tapping in the courtyard
High apartment houses on all four sides. A canvas tent in the middle. A black Orlov trotter with a white blaze, and beside him a retired schoolteacher in a wide-brimmed hat — Wilhelm von Osten, who has spent four years teaching this horse.
Somebody in the crowd calls out a sum. Von Osten writes it on a blackboard. Hans lowers his head, lifts his right forefoot, and begins to tap the flagstones. Tap. Tap. Tap. He stops on the correct number. Every time. He can tell the time, name the value of coins, spell words with a tapped-out alphabet, and pick a tone out of a chord.
Thirteen serious people — a zoo director, a veterinarian, two retired officers, a circus manager, schoolteachers and the psychologist Carl Stumpf — examined Hans and signed a report on 12 September 1904 saying there was no trickery. They were right. There was no trickery. And the horse still could not add.
Everything that goes wrong here is still going wrong today, in hospitals, in cameras, in the models that sort your photographs. Only the horse has been replaced.
This lesson takes about 25 minutes. Everything on this page — every picture, every model, every number in every chart — is computed live in your browser while you watch. Nothing here is a recording.
Act one · the bench
Ask Hans a question
Here is the courtyard, and here is the horse. Pick a sum, ask it, and watch him tap. Then start turning the two switches underneath, one at a time, and keep score. You are about to re-run the most famous experiment in the history of animal psychology, and it takes about ninety seconds.
The trace above is the height of the questioner's head during the last trial, sampled forty times a second. Ask a question and watch it.
What Pfungst actually measured
Oskar Pfungst was an unpaid volunteer in Carl Stumpf's Berlin laboratory — no doctorate, no title. Between 13 October and 29 November 1904 he put a canvas tent up in the courtyard and took the problem apart with two switches, exactly the two you have just been flipping.
| Pfungst's test | Condition | Trials | Hans correct |
|---|---|---|---|
| Reading numbers | Questioner did NOT know the answer | 49 | 8% |
| Reading numbers | Questioner knew the answer | 42 | 98% |
| Addition | Questioner did NOT know the answer | 31 | 3 right |
| Addition | Questioner knew the answer | 31 | 29 right |
| Large blinders, 15 × 15 cm | Questioner NOT seen | 35 | 6% |
| Large blinders, 15 × 15 cm | Questioner seen | 56 | 89% |
Figures read directly from Pfungst, Clever Hans (The Horse of Mr. von Osten), trans. C. L. Rahn, Henry Holt, 1911, chapter II, pp. 35–44. Pfungst's own summary, p. 40: "in the case of procedure with knowledge, 90 to 100% of the responses of the various series were correct, whereas, in those series of procedure without knowledge 10%, at most, of the responses were correct."
Note carefully which pair goes with which switch. The famous 89% and 6% are the blinders figures — could Hans see the questioner. The knowledge switch gives 98% and 8%. Almost every retelling on the internet swaps them.
Act two · swap places
Now you are the horse
Knowing the answer is not the same as feeling it. So stand where Hans stood. You will not be shown the sum. You will be shown a person — and you must tap until you feel it is time to stop. Tap with the button or the space bar.
Press "Begin a trial". Watch the person, not the numbers — there are no numbers.
If you got any of them right without the magnifier, congratulations: you have just done what Hans did, and it did not feel like cheating. It felt like noticing. That is the entire point. Hans was not a fraud and neither are you. He was an extraordinarily good reader of people who had been rewarded, thousands of times, for stopping at a particular moment.
Pfungst found the cue: as the questioner finished writing, he bent his head and trunk very slightly forward — Hans started tapping. When the tapping reached the right number, the questioner made "a slight upward jerk of the head" — Hans stopped. Pfungst was making these movements himself without knowing it. He writes that he only saw his own movements after he had noticed von Osten's, and that other people watching him work with the horse "never perceived them, no matter how closely they observed."
He could not switch the cue off. Bending down to write a note was enough: "Hans would invariably begin to tap when I stooped to jot down some note I wished to make."
Act three · the training floor
The same trick, in silicon
Ninety-five years later, a research group in Berlin — a few kilometres from von Osten's courtyard — pointed an explanation method at a well-regarded image classifier and found the horse again. Literally the horse: the classifier's best evidence for the category "horse" in the PASCAL VOC 2007 dataset was a small copyright tag printed in the lower-left corner of about one fifth of the horse photographs. Paste that tag onto a photograph of a Ferrari and the model calls the Ferrari a horse.
So let's build one. Below, your browser will draw a few hundred tiny 32 × 32 pictures of horses and cars from scratch, stamp a little printer's tag into the lower-left corner of most of the horses, and train a real neural network on them — a two-layer network with hand-written backpropagation, about 16,000 weights, no libraries. Then it will test that network three ways.
A sample of what the network is being shown. Horses on the top row, cars on the bottom. Look at the lower-left corner of each.
Press "Train the network". It takes a second or two.
The Ferrari moment
Here is one car. The model is fairly sure it is a car. Now stamp the tag on it — the tag has nothing to do with cars or horses, it is a printer's mark — and watch the verdict.
Train the network first, then press the button.
Act four · where is it looking?
Cover one square at a time
Pfungst could not ask the horse what he was looking at, so he put blinders on him. We can do something better and cruder: cover a small square of the picture, run the model again, and see how much the verdict moves. Do it for every square and you have a map of the evidence. This is called an occlusion map, and it is the crude honest ancestor of every fancy explanation method in the field.
The picture
Where the evidence is
The two, laid on top of each other
Train the network in the act above first.
—
—
The two pictures above are the average evidence-map of each group, found by clustering the maps of 48 horse pictures into two families. In the real 2019 study this idea is called Spectral Relevance Analysis, and it found four separate strategies hiding inside one "horse" category: look at the horse, look at the rider's hurdles, look at the tag in portrait photographs, look at the tag in landscape photographs.
The measured curve
How reliable does a shortcut have to be?
Not very. Below, your browser trains a fresh network at each tag reliability from 0% to 100% and measures all three numbers. The green line is the one everybody reports in a paper. The orange line is the one that matters when the model leaves the building.
Act five · the ward
The score goes up. The medicine does not.
In 2018 a team led by John Zech tested pneumonia detectors across three hospital systems. At Mount Sinai, 34.2% of the chest radiographs in the study were positive for pneumonia. At the National Institutes of Health, 1.2%. At Indiana, 1.0%.
Radiology technicians clip a small metal token onto the patient, and it shows up in the corner of the image. Portable scanners — wheeled to the bedside of patients too sick to walk to radiology — produce visibly different images. A network trained to name the hospital from the image alone got 99.95% of NIH films right, 99.98% of Mount Sinai's, and 95.59% of Indiana's. Within Mount Sinai, it told the inpatient floor from the emergency department with 100% accuracy.
Now watch what that does to a score. Below is a small simulated hospital system with the real prevalences. Each patient has a genuine lung signal, worth something but not everything, and a token that says which hospital they are in. Train a model with the token visible, and with it hidden, and compare four numbers.
This is a simulation built to reproduce the mechanism, not the study's data. The real published figures: a model trained on NIH scored 0.750 on NIH's own held-out films and 0.695 at Mount Sinai; a model trained on both together scored 0.931 on the pooled test set — and that 0.931 is the confound talking, not the lungs.
A single leaderboard number cannot tell you whether your model is reading the lungs or reading the hospital. Nothing about a good score rules out a shortcut — a shortcut is defined as the thing that gets a good score.
The receipts
Six times a machine was Clever Hans
Every one of these is documented, published and numbered. Not one of them involved a broken model. Each model was working perfectly — on the problem it had actually been given, which was not the problem anyone meant to give it.
Act six · a story about a story
The tank that never was
There is one story that every machine-learning course tells. The army trained a neural network to spot tanks hidden in a forest. It worked perfectly in the lab and failed completely in the field, because all the tank photographs had been taken on a cloudy day and all the empty-forest photographs on a sunny one. The network had learned to detect the weather.
It is a wonderful story. It is almost certainly not true. Click along the chain below and see how far back it goes.
The trail ends at an email. In 2013, Edward Fredkin wrote that the story came from him: he had been in the audience at a talk in Los Angeles around 1963 and had raised the possibility that the network might have "merely trained itself to recognize the difference between a bright picture and a dim picture." A question from the floor. Never tested, never answered — and over fifty years it hardened into a documented failure with dates and photo counts attached.
Meanwhile the real anecdote from that era is better, and it is first-hand. Marvin Minsky described a perceptron trained to tell chorales written by Bach from chorales written by conservatory students. It succeeded. Then he looked: one of the two sets of scores had a single whole note in the lower left-hand corner of each page. The machine was reading the corner.
A legend survived half a century without a shred of evidence because everyone who heard it recognised something true. The phenomenon never needed the tank. It has hospitals, dermatology clinics, camera traps and a horse.
Where this page is thinner than the truth
Six honest wobbles
One hundred and twenty years
From a courtyard to a chest X-ray
Click any stop.
The examination
Eight questions, and no cues from the examiner
Wrong answers are allowed and are answered honestly. There is no head-jerk hidden anywhere on this page.