Change every pixel by less than 1% and a top image AI swears a panda is a gibbon. Find out why, then do it yourself.
You are a photograph of a panda, and every one of your pixels is a brightness number. You have been shown to GoogLeNet, a network trained on more than a million photographs. It decides you are a panda, with 57.7% confidence: fairly sure, for a guess among a thousand possible answers.
Now a researcher does something odd. He does not ask the network what do you see. He asks it a different question, the question this whole lesson turns on:
The network answers, because that question uses the same arithmetic it runs all day while learning. It hands back a number for every pixel (brighten this one a hair, darken that one), and the researcher keeps only the direction, up or down. Then he nudges every pixel by 0.007, which the paper notes is about the smallest step an 8-bit image can store.
Put the before and after side by side and no person can tell them apart. Feed the after to GoogLeNet and it says gibbon, with 99.3% confidence. A picture changed on purpose to fool a machine like this is called an adversarial example.
Not "maybe a gibbon". Not "a panda, possibly". More certain that you are a gibbon than it ever was that you were a panda.
That is Figure 1 of Explaining and Harnessing Adversarial Examples by Ian Goodfellow, Jonathon Shlens and Christian Szegedy, posted on 20 December 2014. It became one of the most reproduced pictures in machine learning. On this page you will do the same thing yourself, to a real network trained in your browser.
Nothing here is pre-recorded. Your browser draws 2,000 small pictures of four things (a panda, a gibbon, a teapot and an ostrich) at random sizes, angles and positions, then trains a real neural network on them from scratch. It takes a few seconds, and every attack after this targets that network.
24 × 24 pixels each. Grey values from 0 to 1. Drawn by this page, not photographed.
Accuracy on pictures it studied (blue) and on pictures it has never seen (orange), measured after every pass.
A neural network learns by asking, millions of times, if I made this weight a little bigger, would I be more wrong or less wrong? The answer is a gradient: a set of tiny arrows, one per weight, all pointing toward more wrong. Training walks the other way, downhill, one small step at a time.
An adversarial attack uses the same machinery, aimed at the picture instead of the weights. You ask which way should each pixel move to make you more wrong? and take one step uphill. That is the whole attack: a few lines of code that were already inside every training loop.
x_adversarial = x + ε · sign( ∇x J(θ, x, y) )
Read it aloud: the picture, plus a tiny amount, times the direction of the slope of the wrongness with respect to the picture. Goodfellow called it the fast gradient sign method, and "fast" is the point — one step, one pass, no searching.
The bench below hands you the dial. The middle panel is the perturbation itself, amplified so you can see it at all; the left is the photograph; the right is what the network actually receives.
Pick a picture. Turn the dial. Watch the network change its mind about something that has not visibly changed.
Magenta = brighten this pixel. Green = darken it. Each by exactly the same tiny amount.
“Make everything an ostrich” is not our joke. In the 2013 paper that started this, Figure 5 shows a column of unrelated photographs, every one pushed until AlexNet called it ostrich, Struthio camelus.
Each round, your network gets a picture with a small change added. You have 8 seconds to call it. The answer comes from really running the attack.
Will the network be fooled, or still name the picture correctly?
The first explanation everyone reaches for is that neural networks are weird, twisty, deeply non-linear things, and that adversarial examples live in strange little folds of that twistiness. Goodfellow's paper argued almost the opposite, and this is its real contribution:
Here is the arithmetic. A neuron adds up its inputs, each multiplied by a weight. Push every input by a tiny ε in the direction its own weight prefers, and the neuron's total moves by ε times the sum of all the weight sizes.
change = ε · Σ|wi| = ε · n · mean|w|
That sum has an n in it: the number of pixels. Your eye checks one pixel at a time and sees ε, but the neuron adds them all up and feels about ε × n. In a 224 × 224 colour photo, the size GoogLeNet reads, n is 150,528, so the attack is not slipping through a crack but walking through a door that widens with every pixel.
Two dials: how far each pixel moves, and how many pixels there are. Watch the two bars: what your eye is asked to notice and what one neuron actually feels.
Challenge: keep ε at 0.007 or less and drag n until one neuron feels a total change of at least 1.
Train the flat network (no hidden layer) and its weights are pictures: one per class, 576 numbers each. Magenta means “brightness here votes for me”; green means “brightness here votes against me”. Compare the panda-to-ostrich attack with the difference between two of these pictures: they turn out to be the same thing.
A trick that only works on one particular network is a curiosity. You would need the network's insides to build it, and if you have the insides you probably have easier ways to cause trouble.
But that is not what happens. In December 2013, seven researchers at Google, New York University and the University of Montreal (Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow and Rob Fergus) found that a picture built to fool one network often fools other networks too. It worked across different designs, different random starts, and even networks trained on a different part of the data.
So the flaw is not a private quirk of one machine. It is a blind spot that machines trained on similar data tend to share. And a shared blind spot lets you attack a system whose insides you have never seen.
Below, your browser trains three more networks: one with a different random start, one with a different architecture, and one on a disjoint half of the pictures. Then it attacks only the first network — and hands the results to the other three.
The attack never touches networks B, C or D. It does not know they exist.
If you can walk uphill from a photograph, you can walk uphill from anything. In 2015, Anh Nguyen, Jason Yosinski and Jeff Clune started from random static and climbed, producing images that top networks labelled with 99.99% confidence. To a person they look like static or wallpaper; to the network they were a starfish, a baseball, a remote control.
This is the other side of the same coin. Something that is obviously X can be made to read as Y, and something that is obviously nothing can be made to read as X, more confidently than any real photo. Both happen because “how confident the network is” and “how much the picture looks like the thing” are two different numbers.
Start from random noise. Climb toward a class, one small step at a time. Nothing constrains the picture to look like anything.
So far, all of this happens inside a file. Real cameras wobble, light changes, and a sign is seen from many angles, so a nudge tuned for one exact view tends to vanish when the view moves. For a few years that was the comforting answer: it will never survive the real world.
It did. In 2017, Kevin Eykholt, Ivan Evtimov and colleagues stuck black-and-white stickers on a real stop sign; one pattern even spelled LOVE and HATE, like graffiti. Their best stickers made the sign reader see Speed Limit 45 in 100% of lab photos and in 84.8% of video frames filmed from a moving car.
The same year, four MIT students (Anish Athalye, Logan Engstrom, Andrew Ilyas and Kevin Kwok) 3D-printed a turtle you could hold and turn over. Photographed 100 times from different angles, it was called a rifle by Google's InceptionV3 in 82% of the shots and a turtle in only 2%. Their 3D-printed baseball was called an espresso 59% of the time.
The trick has a plain name and a lovely idea: Expectation Over Transformation (EOT). Instead of asking which nudge fools the network from here, you ask which nudge fools it on average across many angles, distances and lighting. Below, watch the one-view attack fall apart first, then try the fix.
Your perturbation is now stuck to the object, not to the file. The camera moves: the picture is shifted, scaled and rotated at random, and then classified. Build the attack from one fixed viewpoint, or from forty random ones, and see which one still works when the car drives past.
Once the problem was public, fixes poured in: round the pixels off, blur the image, add randomness so the attacker can't aim. Each one made the standard attack's numbers collapse. For a while, each was published as a defence.
In February 2018, Anish Athalye, Nicholas Carlini and David Wagner studied nine such defences from one conference. Seven relied on obfuscated gradients: they hid the slope so the attacker's compass spun, without removing the weakness. The team broke six completely and one partially.
One thing does help, and it is the boring thing: put adversarial examples into the training set. Aleksander Mądry's group trained against a strong many-step attack, and on CIFAR-10 their model scored 87.3% on clean pictures and 45.8% under that attack. That is real progress with a real price: their normally trained model scored 95.2% on clean pictures, so robustness cost about eight points.
Run both stories below in under a minute. Try the cheap defence first and watch its numbers look wonderful. Then attack it with someone whose compass doesn't spin.
Three networks, same pictures, same attack: one trained normally, one with a cheap trick bolted on the front, and one trained on its own nightmares (adversarial examples).
One caution about the third bar. Telling four cartoon shapes apart is an easy problem, so here adversarial training is almost free. On real photos it is not: Mądry's CIFAR-10 model gave up about eight points of clean accuracy (95.2% → 87.3%), and robust models since have paid some version of that bill.
For five years everyone called these things bugs. Then in 2019 Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran and Aleksander Mądry proposed something far more unsettling: they are not bugs. They are features.
Here is the argument. A photo of a cat holds patterns that help say “cat”: some you can see, like whiskers and ears, and some are faint patterns spread over thousands of pixels that you can't see but that really do predict “cat”. The network uses everything that works, and the invisible patterns are exactly what a tiny nudge can rewrite, so an adversarial picture is one where the hidden evidence says “gibbon” while the visible evidence still says “panda”.
They tested it with an experiment that feels like a magic trick. Attack every training picture toward a new label, then relabel each picture with the label the attack aimed at, so to human eyes the whole training set is wrong. Train a brand-new network on it, then test it on ordinary, untouched photos.
It should score at chance, but it doesn't: on CIFAR-10 it scored 43.7%, where guessing gets 10%. The only thing that survived the relabelling was the invisible evidence, and on its own that evidence works on real photos.
You can run it here. It is the same experiment, much smaller, and it may or may not work on a network this size: report what you actually get, not what you hoped for.
Every training picture is attacked toward a new class, then stamped with that new class. A human marking this notebook would give it zero. We then train a fresh network on it and test on clean pictures it has never seen.
Top row: the picture. Bottom row: the label it has been given.
Two honest footnotes. We keep only pictures where the targeted attack actually landed, and the read-out says how many that was. The right-hand bar is the control (same random labels, same size of change, but random noise instead of an aimed nudge), so if the left bar beats it, the information was carried by the aim, not the ink.
1. These are not photographs, and this is not GoogLeNet. Your browser drew every picture on this page from shapes and randomness, and the network you attacked has a few tens of thousands of numbers in it. GoogLeNet had millions and had studied 1.2 million real photographs. Everything you saw is real — real gradients, real backpropagation, real attacks — but "it worked here" is a demonstration of the mechanism, not a measurement of the world.
2. "Imperceptible" is doing a lot of work. ε = 0.007 is genuinely invisible. ε = 0.15, which you can dial up above, is not — you can see the grain. Papers measure "small" in at least three incompatible ways (largest single-pixel change, total energy, number of pixels touched), and a result in one is not a result in another. Whenever someone says an attack is imperceptible, ask: in which norm, and at what size?
3. The stop-sign result is contested. In 2017, Jiajun Lu, Hussein Sibai, Evan Fabry and David Forsyth published a paper titled “NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles”. They argued that a detector that sees a sign at many distances as a car approaches is much harder to fool than a single-photo classifier. Physical attacks are real research, but the jump from “a sign reader was fooled in tests” to “self-driving cars can be crashed with a sticker” goes beyond the evidence.
4. Nobody has fixed it. Not in more than a decade. In 2024 the best reported CIFAR-10 model (Bartoldson and colleagues) reached 93.7% on clean images and 73.7% under the standard AutoAttack test. That beats 45.8% in 2017, but it is far from solved, on tiny 32 × 32 pictures with only ten classes.
5. "Not bugs, features" is a hypothesis, not a verdict. The 2019 paper was published alongside a whole discussion issue of responses arguing about what its experiments do and do not show. It is a serious, well-evidenced idea. It is not settled science, and this page presents it as the interesting argument it is.
6. This did not begin in 2013. Spam filters were being gamed deliberately, and studied academically, long before deep learning: Nilesh Dalvi and colleagues framed it as a game between classifier and adversary in 2004, and in September 2013 — three months before the famous neural-network paper — Battista Biggio's group published gradient-based evasion attacks on support vector machines and malicious PDF detectors. Deep networks made the problem vivid. They did not invent it.
Eight questions. Wrong answers are, as ever, more fun than right ones.
You need squared paper, a pencil and one other person. No computer.
Build the detector. Draw a 5 × 5 grid and write +1 or −1 in each square: these are your network's weights. Draw a second 5 × 5 grid, the picture, and write a number from 1 to 9 in each square.
Run it. Multiply each picture square by the weight in the same spot and add all 25 results. Above zero means YES, below zero means NO; try pictures until you get a confident YES.
Attack it. You may change every picture square, but only by one, up or down: that is your ε. Where the weight is +1, go down by one; where it is −1, go up by one; then add up the total again.
The score drops by exactly 25, one per square, though no square moved more than a tenth of its range. That is the fast gradient sign method done with a pencil, and the 25 is the n. On a 10 × 10 grid, the same trick moves the score by 100.
Boss level. Challenge the other person to design a detector that a change of one per square cannot flip, and give them ten minutes. Then tell them, gently, that researchers have been working on that exact problem since 2013.
The papers and pages the facts on this page come from.