How fifty thousand strangers, a bedroom computer with two graphics cards, and a very old idea taught machines to see — and why the room went quiet.
Before any machine could see anything, a person had to look at a picture and say what was in it. About fifty thousand people did that, from 167 countries, for roughly two and a half years.
It is late. The page loads slowly. A stranger in New Jersey needs to know whether this photograph contains a mushroom. Not a picture of the word. Not something mushroom-ish. A mushroom.
Fei-Fei Li had an idea most of her colleagues thought was a waste of a career. In 2006, as a new professor, she decided that computer vision's problem was not its algorithms but its diet. Machines were being fed a few thousand pictures and asked to understand a world of millions. So she went looking for a list of everything — and found one: WordNet, a map of over 145,000 English words built by the Princeton psychologist George A. Miller, in which words nest inside words: Dalmatian inside dog inside mammal inside thing.
Her plan: hang photographs on every hook of that map. Grant reviewers turned her down. One comment said it was shameful that Princeton would research this topic. Colleagues asked the obvious question: if you can't even do one object well, why would you do thousands?
Then came the arithmetic. Search engines could fetch candidate pictures, but roughly three out of four were wrong — for the mammal branch, ImageNet measured the search results as only 26% accurate. Every single image needed a human to check it. Li's own slide from 2010 does the sum: 40,000 synsets × 10,000 candidates × 3 passes ÷ 2 images per second = 600,000,000 seconds ≈ 19 years. At the pace her undergraduates were actually working, a journalist later reported the estimate as 90 years.
The solution arrived in a hallway. A graduate student asked whether she had heard of Amazon Mechanical Turk — a website where anyone, anywhere, could be paid small amounts to do small tasks. Suddenly 19 years of looking could be done by fifty thousand pairs of eyes at once.
Take a shift. Here is the page as it would have appeared to you.
Notice what you were not doing. You were not teaching a machine what a mushroom is. You were building a very long list of the form [picture] → [word], one row at a time. By the end there were 14,197,122 rows across 21,841 categories — and a competition subset of 1,000 categories and about 1.2 million training pictures, released each year as a challenge: ILSVRC.
In 2009 the paper describing all this got a poster at CVPR — not a talk — and, in Li's telling, little fanfare. The list sat there for three years, waiting for someone to be hungry enough for it.
A neural network does not look at a picture. It slides tiny stencils across it and adds things up. Here is one stencil. You are holding it.
You are a filter: a 3×3 grid of numbers, no memory, no idea what a mushroom is. A patch of nine brightnesses arrives. You multiply, you add, you shout one number. Then you slide one pixel to the right and do it again, forever.
Drag any square up or down to change its number. Watch the picture on the right change. Hover the input to see the exact multiplication happening under your stencil — this is the whole operation, no metaphor: out(x,y) = Σ k(i,j) · in(x+i, y+j).
The first layer of the 2012 network held 96 of these stencils, each 11×11 pixels across three colour channels, sliding in jumps of 4 pixels. Stack layers of them — five convolutional layers, then three fully connected ones — and you get 60 million numbers and about 650,000 artificial neurons.
Here is the part that should stop you. Nobody chose those 60 million numbers. No engineer sat down and typed in an edge detector. The network was handed pictures, allowed to guess, told how wrong it was — and it edited itself. Sixty million times over, ninety times through 1.2 million pictures.
This is a real neural network — written in a few dozen lines running in your browser, no libraries, no pretending. Press Train and watch it change its mind.
You are now the machine on the desk in Alex Krizhevsky's bedroom. Two fans howl. Every guess you make is scored, and every score is pushed backwards through you, nudging each number a little in the direction that would have hurt less.
Each dot is one training example. The background is the network's opinion about every point on the map: amber means "class A", pale blue means "class B". Drag dots. Add your own. Then poison the labels and watch what happens to a machine that trusts its teachers completely.
That is gradient descent, and it is the entire trick. Guess, measure the error, ask each number "which way should you move to make that error smaller?", take a small step, repeat. The learning rate is how big the step is. Too small and you crawl; too big and you fall over. Krizhevsky started at 0.01 and divided by ten whenever progress stalled — three times, over about 90 passes through the data.
And the poison slider is not a party trick. A network has no way to doubt its labels. It will faithfully learn whatever wrongness you hand it — which is exactly why fifty thousand people were checking mushrooms in Act I.
The workshop was in Florence, attached to a computer-vision conference. Fei-Fei Li opened the session at 15:30. She had almost stayed home in California with her baby. Then she saw the scores.
You are standing at the back, because there are no chairs left. Three teams present. The first two have improved on last year the way this field always improves on last year: by about a point. At 17:05 the third team's slot begins.
The benchmark was top-5 error: the model offers five guesses for each photograph, and it fails only if the right answer is in none of them. In 2010 the winner failed 28.2% of the time. In 2011, 25.8%. Progress was real and slow, and everyone in the room knew roughly what 2012 should look like.
Press play. Reveal the years one at a time, the way the room got them.
The entry was called SuperVision: Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, University of Toronto. Its method line on the official results page reads, in full: "Large, deep convolutional neural network trained on raw RGB pixel values." Raw pixels. No hand-built features. The best of everyone else — the ISI team from the University of Tokyo — scored 26.2%.
Both figures are correct and they are not the same claim. Almost every retelling of this story picks one and drops the footnote. The margin over second place was 10.9 percentage points — in a field that had been moving one point a year.
As Fei-Fei Li later recounted it, Yann LeCun stood up in that room and called the result "an unequivocal turning point in the history of computer vision. This is proof." Five months later Google bought the three-person company the authors had formed. Hinton's own summary of the collaboration is the best sentence in the whole affair: "Ilya thought we should do it, Alex made it work, and I got the Nobel prize."
WordNet contained every noun, so ImageNet inherited every noun. Including the nouns you cannot photograph.
The map had a branch for person, and under it more than 2,800 categories. Some are things a camera can plainly record: a flutist is a person holding a flute. Others are judgements, insults, or claims about someone's inner life and history that no pixel can verify — and yet each one had thousands of photographs of real people filed underneath it, scraped from the web, labelled by strangers.
Sort these eight real-style labels. Which could a single photograph honestly establish?
In 2019 the artist Trevor Paglen and the researcher Kate Crawford built ImageNet Roulette, which let anyone upload a photo and see which person-category the dataset would file them under. The results were, predictably and then shockingly, ugly. Their essay's line is the one worth memorising: "Images do not describe themselves."
The dataset's own team audited the branch and published the numbers: of 2,832 person categories, 1,593 were flagged as unsafe — offensive regardless of context — covering 600,040 images. Of the rest, only 158 were judged genuinely visual; 1,081 more were words you cannot see, covering another 443,547 pictures. Those images came out. Later releases blurred faces: in the 1,000-category challenge set, 562,626 faces were found across 243,198 photographs — 17% of a dataset that was never about faces at all.
Keep both halves of this in your head at once. The list was the breakthrough, and the list was the problem. A machine that learns from what we collected learns what we collected — carelessness included.
Every claim on this page was checked against the original papers, the official results pages and the workshop programme. Here is what fell apart under checking.
You need a handful of dried beans or coins, paper, and a pencil. Draw two big overlapping circles on the paper and label them A and B. Have a friend secretly invent a rule — "beans on the left half are A", say, or something sneakier like "beans touching a pencil line are A".
Now play the machine. Place a bean. Guess A or B. Your friend says only "yes" or "no" — never the rule. Draw your current best guess at the boundary as a light pencil line, and after each "no", rub it out and move it a little way toward the bean you got wrong. Not all the way — a little. That "little" is the learning rate.
Two things will happen, and both are the lesson. You will find the rule without ever being told it. And if your friend lies to you on one bean in five, your line will end up in a place that is nobody's rule at all. Congratulations: you have built a classifier and you have poisoned a dataset.
No half marks, but generous laughter at wrong answers.