AI & Machine Learning · Daily Learning Experience
A sentence at dusk, one ambiguous little word, and the question that rebuilt artificial intelligence: who do I mean?
Act one · The word with no meaning
You are the word “it”. You have no colour, no shape, no smell and no meaning of your own. You are a pointing finger with the room lights off.
The sentence you have landed in is famous among people who build language machines. It goes: “The fox didn’t cross the street because it was too tired.” Every reader over the age of six knows instantly who was tired. The fox, obviously. Streets do not get tired.
Now change one word. “The fox didn’t cross the street because it was too wide.” You did not move. Your letters did not change. But you now mean something completely different — and you found that out from a word standing three places away from you, on the far side of the sentence.
That is the whole problem. A word’s meaning is not printed inside the word. It has to be fetched from the neighbours. For about thirty years, machines solved this by passing meaning down the sentence hand to hand, like a bucket brigade, one word at a time. It was slow, and buckets got dropped.
Then on 12 June 2017 at 17:57 UTC, eight researchers at Google posted a paper to arXiv with a title borrowed from a Beatles song. Their idea was almost rude in its simplicity: stop passing the bucket. Let every word look at every other word, all at once, and let each word decide for itself who is worth looking at.
They called the operation attention. They called the machine a Transformer. Almost every AI system you have heard of since is made of these.
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin, “Attention Is All You Need”, arXiv:1706.03762, submitted 12 June 2017; presented at NIPS 2017. All eight are marked “Equal contributor. Listing order is random.”
Act two · The query desk
Attention is a question and a room full of name badges. Each word wears a key — a badge saying what it is. Each word holds a value — what it would hand over if asked. And each word can send out a query — what it is looking for. That’s it. That is the whole mechanism.
You are “it”, so you get to ask. Below, four dials build your query. Turn them up and you are asking loudly for that kind of word; turn them below zero and you are actively avoiding it. Your query is compared with every badge in the sentence by multiplying the numbers and adding them up — a dot product. Big number means good match.
These four dimensions have names so you can read them. A real Transformer uses 64 of them per head, and nobody named any of them — the model invented them while it was learning. We’ll train a real one in Act four.
The bar under each word is how much of your attention it gets. The percentages always add to 100 — that is what softmax does: it turns any list of scores into a set of shares that sum to one.
This is the smallest, oddest detail in the paper, and it matters. The more dimensions you compare, the bigger the dot products get — their spread grows with dk. Feed huge numbers into softmax and it stops being a set of shares and becomes an all-or-nothing switch: one word gets 100%, everyone else gets zero. A switch has no slope, and a thing with no slope cannot be learned from. Dividing by √dk holds the numbers in a range where the model can still feel which way to move.
Same four words, same comparison, scaled up to dk dimensions.
Act three · One word changes everything
Pick a creature, a place and a last word. The street above re-lights itself, and the glowing cable from “it” swings to whoever the sentence now means. Nothing in the middle of the sentence changed. The last word decided.
Game · think like a query
Which word gets the most attention?
Act four · The training floor
Everything above used badges we wrote so you could read them. That is cheating. Here is a real model — two attention layers, 45 words in its vocabulary, 1,488 numbers, every one of them currently random. It knows nothing. It is going to learn this, right now, in your browser, from 140 example sentences and nothing else.
Watch what it has to work out. Layer one asks a first question: what is being said about me? Layer two asks the second: so who am I? The two questions have to be asked in that order — you cannot know which noun you mean until you know what the sentence claims. That is why real Transformers are stacked six, twelve, ninety-six layers deep. In the original paper the head that resolved “its” to “Law” lived in layer five of six.
How wrong it is, on average, per sentence. Lower is better. Zero would be perfect.
These four sentences were deliberately kept out of the training set. Every individual word appears somewhere in training, but never in this combination. If the model had merely memorised, it would fail here.
Act five · Where are you?
Here is a strange hole in the idea. If every word looks at every other word all at once, then nothing in the machine knows what order the words came in. “Dog bites man” and “man bites dog” are, to raw attention, the same soup.
The fix is one of the prettiest tricks in the paper. Before the words go in, each position is given a fingerprint made of sine and cosine waves at many different frequencies — fast waves for the fine detail of nearby positions, slow waves for the broad sense of early-versus-late. Add the fingerprint to the word. Now “the” at position 1 and “the” at position 6 are no longer identical twins.
Every square in the picture is computed from that formula live — no image was pasted in. Read a row left to right and you are reading one position’s fingerprint. Read a column top to bottom and you are watching one wave tick.
13 positions down, 16 dimensions across. Dark = high, pale = low.
Act six · Eight questions at once
A single attention head can only ask one question per word. But a sentence needs several answers at the same time: who is the subject, what does this pronoun mean, which adjective belongs to which noun. So the paper runs eight heads side by side, each with its own small 64-dimensional space, and glues the answers together. Click a head to see the question it is asking.
The sentence below is the real one from Figure 4 of the paper. The authors found a head, in layer five of six, that had taught itself to resolve pronouns — and its attention for the word “its” was, in their words, “very sharp.”
Figure 4 caption, verbatim: “Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution… Note that the attentions are very sharp for this word.” Sentence: “The Law will never be perfect, but its application should be just - this is what we are missing, in my opinion.”
Honesty note: the three heads here were hand-built so their questions are readable. The paper’s heads were not designed — they were discovered, after training, by people staring at heat maps and being surprised.
Act seven · Why it ate the world
Attention was not new in 2017. What was new was throwing away everything else. And the reason that mattered has almost nothing to do with cleverness and everything to do with time.
The paper’s Table 1: “maximum path length” between any two positions.
The base model in the paper was small by today’s standards: 6 layers, 512 dimensions, 8 heads of 64 each, 65 million numbers. It trained on 4.5 million English–German sentence pairs in twelve hours on eight NVIDIA P100 graphics cards, and beat every previous system on the benchmark. The bigger version took 3.5 days.
Speed was the whole point. A bucket brigade cannot be hurried — word 40 has to wait for word 39. Attention has no queue, so a room full of graphics cards can work on the entire sentence in one breath. That is why the idea spread: not because it thought better, but because you could afford to make it enormous.
The bill comes due elsewhere. Every word looks at every word, so a sentence of n words needs n² comparisons. Double the length, quadruple the work. That single fact is why “context length” is the thing AI companies keep bragging about — it is genuinely expensive.
Six things this page is being careful about
The four named dimensions in Act two are a teaching fiction. A real head has 64 dimensions and none of them means “creature-ish”. They mean whatever combination the gradient happened to find useful, which is usually not describable in words at all.
It is very tempting to say “the model looked at ‘fox’, so ‘fox’ is why it answered”. Researchers fought about exactly this. Jain & Wallace argued in 2019 that you can often scramble attention weights and get the same prediction (“Attention is not Explanation”, NAACL 2019); Wiegreffe & Pinter replied the same year with “Attention is not not Explanation” (EMNLP 2019). Treat a heat map as a hint, not a confession.
To make Act four fast and legible, we train it directly on “point at the fox” — the attention itself is supervised. Real Transformers are never given that instruction. They are only ever asked to predict the next word, and the pointing behaviour appears on its own as a side effect. That it appears at all is one of the genuinely surprising facts of the field.
The title is a joke on a Beatles song, suggested a couple of nights before the deadline. The actual model also needs feed-forward blocks, residual connections, layer normalisation and those positional fingerprints. In the base model, each layer’s feed-forward block holds roughly 2.1 million numbers against attention’s 1.05 million — twice as many. The famous part is the smaller half.
Attention arrives in September 2014, in Bahdanau, Cho & Bengio’s “Neural Machine Translation by Jointly Learning to Align and Translate” (arXiv:1409.0473), where it was bolted onto a recurrent network. Luong and colleagues sharpened it in 2015. Parikh and colleagues — including Uszkoreit — dropped recurrence for a different task in 2016. The 2017 paper’s contribution was the nerve to remove everything else.
Press “new brain” in Act four a few times. Every run reaches the same answers; the internal picture keeps changing. Sometimes layer one stares straight at the adjective; sometimes it parks on the word “because” and picks up what it needs sideways. If a model can be right in many shapes, then any story you tell about one particular shape is a story about this run, not about understanding.
The long fuse
The examination
Instant marking, and the wrong answers are more fun than the right ones.
Do this at the kitchen table
You need paper, a pencil, and at least two people. Write a sentence with a slippery “it” in it — “I couldn’t put the trophy in the suitcase because it was too big.” Everyone draws an arrow from “it” to the word they think it means. Compare. Now change one single word — big to small — and vote again. Watch every arrow move.
Boss level. Give everyone two index cards. On the KEY card, each person writes three words describing what they are (“loud, hungry, eight”). On the QUERY card, they write three words describing what they’re looking for (“quiet, tall, funny”). Now shuffle the keys onto the table face up. Each person scores every key by counting matching or related words — that count is a dot product. Highest score wins your attention. Hand your value card to the winner. You have just run one attention head with human beings as the vectors.
Second boss level. Do the pointing game again, but read the sentence out loud one word at a time, and make everyone commit to an arrow before the last word is spoken. Notice how badly you want to wait. That impatience is exactly the problem the Transformer solved.
Where the facts come from