Daily Learning Experience · AI & Machine Learning
You are a machine listening to a sentence. Somewhere in this room a person has just spoken. You will never hear a word — only a hundred numbers a second. This is the story of how machines learned to guess, and of the afternoon it turned out that guessing was the whole job.
Act one · Murray Hill, New Jersey, 1952
You are AUDREY. You stand against the wall of a basement laboratory at Bell Telephone Laboratories: a rack of relays, vacuum tubes and a little bank of neon lamps, one for each digit. A man leans into the microphone and says a number. You have about a second to light the right lamp. You do not know what a number is. You do not know what a word is. What reaches you is a wobble in a wire.
Audrey — the Automatic Digit Recognizer — was built by K. H. Davis, R. Biddulph and S. Balashek and described in the Journal of the Acoustical Society of America in 1952. She recognised the spoken digits zero to nine at between 97 and 99 percent accuracy. There is a catch, and it is the catch that runs through this whole story: that accuracy was for one speaker at a time, after the machine had been tuned to that speaker's voice. Give Audrey a stranger and she fell apart.
Here is the first thing nobody tells you about speech. Look at a page of writing and the words are already separated for you — that is what the little gaps are for. Sound has no gaps. When you say "ice cream," the air does not stop between "ice" and "cream." The whole utterance is one continuous smear, and every sound is bent out of shape by whatever is on either side of it. The machine's very first problem is not "which word is this?" It is "where does anything begin?"
Nothing here is a recording. Every sample is generated in your browser right now by a formant synthesiser — a buzzing "voice box" pushed through three tuned resonators, the same physics as your throat and mouth. The picture underneath is a real spectrogram, computed by a Fourier transform written from scratch on this page.
The pressure wave — what a microphone feels
The spectrogram — energy (dark→bright) at each frequency, 100 frames a second
Drag across either picture to place the scissors. Try to cut exactly between the two words — then look at what you actually cut.
In 1947, three Bell Labs researchers — Ralph Potter, George Koenig and Harriet Green Kopp — published Visible Speech, describing a machine that burned the shape of a sound onto a moving strip of paper. It was built partly in the hope that deaf people might learn to read speech off the page. That did not really work. What it did instead was give engineers, for the first time, something to look at. Every idea in this lesson depends on the fact that a sound can be turned into a picture, and a picture into a column of numbers.
Act two · the shape of a vowel
Your throat makes a buzz. Your mouth is a tube with lumps in it, and a tube with lumps in it lets some frequencies through loudly and swallows the rest. The loud ones are called formants. Astonishingly, the first two — F1 and F2 — are almost enough to say which vowel you just made. Move your tongue down, F1 goes up. Move your tongue forward, F2 goes up. That is a vowel: two numbers and a tongue.
In the same journal, in the same year as Audrey, Gordon Peterson and Harold Barney published the measurement everyone still uses. They recorded 76 speakers — 33 men, 28 women and 15 children — each saying ten words twice: heed, hid, head, had, hod, hawed, hood, who'd, hud, heard. That is 1,520 vowels, measured by hand. Then they played the recordings to listeners and asked what they heard.
Drag the two sliders to move your tongue. The synthesiser rebuilds the sound from your numbers and the spectrum below is measured back out of it, so what you see is what you would hear.
The spectrum of your vowel (measured, not drawn)
Where you are on the 1952 vowel map
Slide "Speaker" up and every formant moves together — that is what happens when the same vowel comes out of a smaller head. The vowel has not changed. The numbers have. This one fact broke speech recognition for thirty years.
Average first and second formant frequencies for ten American English vowels, measured by Peterson & Barney from 76 speakers. Each vowel appears three times — once per group of speakers.
Look at the children's cloud against the men's. A child's /i/ can sit almost exactly where a man's /ɪ/ sits. No rule about frequencies will ever separate those two, because there is nothing there to separate. The information that tells them apart is not in the vowel at all — it is in the rest of the voice, and in the rest of the sentence.
Act three · the trellis
You are the decoder now. In front of you is a grid. Across the top, one column for every hundredth of a second of sound. Down the side, one row for every little piece of every sound a mouth can make. Your job is to draw one line from the top-left corner to the bottom-right — a line that says this sound started here, and this one here. There are more possible lines than there are atoms in the observable universe. You have to pick the best one, and you have about a second.
The idea that rescued speech recognition came from somewhere else entirely. In 1967, Andrew Viterbi published an algorithm for decoding convolutional codes in radio transmission — Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory 13(2), 260–269. It has nothing to do with speech. What it has is a trick so simple it feels like cheating.
Here it is. Suppose two different paths arrive at the same box in the grid at the same moment. Whatever happens after that box happens the same way for both of them — the future does not care how you got here. So the worse of the two paths can never win, no matter what comes next. Throw it away. Now, instead of counting every path, you keep exactly one running best answer per box, and the work stops exploding.
This is a real Viterbi decode of the sound you heard in Act one, against a real acoustic model. Each row is one third of one speech sound; each column is ten milliseconds. Brightness is how well that frame matches that piece. The gold line is the surviving best path.
Act four · the sentence the sound cannot decide
Say "ice cream" out loud. Now say "I scream." If you said them naturally, you made exactly the same sounds in exactly the same order. Not similar. The same. There is no acoustic measurement, no microphone, no amount of computing power that can tell those two apart, because the difference is not in the air. It was never in the air.
This is the moment the field turned. Around 1972 a communications engineer named Frederick Jelinek took over the continuous speech recognition group at IBM's Thomas J. Watson Research Center, and imported an idea from radio: treat speech as a message that has been sent through a noisy channel. Somebody thought of a sentence. It got mangled on the way out — through a mouth, a room, a telephone line. Do not ask "what sound is this?" Ask: of all the sentences a person might have meant, which one is most likely to have come out looking like this?
Two beliefs, added together. On the left, how well the sentence explains the sound. On the right, how likely the sentence was before anyone opened their mouth — that is the language model. λ (lambda) is how much you trust the second one. Real recognisers really do have this dial, and they really do turn it up.
Everything below is computed live: the sound is synthesised, the features are extracted, and every candidate sentence is scored by its own Viterbi pass. Turn the noise up until the machine gets it wrong. Then turn λ up and watch the language model catch it.
The language model has read nothing but the forty short lines below. It counts which word follows which, and that is all it knows about the world. Add a line of your own and everything above recomputes.
Try adding out i scream six times and watch a machine change its mind about the world. This is, in miniature, exactly how a language model is made: no rules, no grammar, no meaning. Counting.
The same utterance decoded at every noise level, measured live on this page: the share of candidate sentences the machine gets right with λ turned off, and with λ set to 10.
Act five · the fifty-year climb
In 1971, a study group chaired by Allen Newell handed ARPA a specification. The system should accept continuous speech, from many cooperative speakers of general American dialect, in a quiet room, over a good microphone, with a vocabulary of about 1,000 words and a highly artificial syntax, tolerating less than 10 percent semantic error, in a few times real time — and it should exist by 1976.
In 1976 a system called HARPY, built by Bruce Lowerre with Raj Reddy at Carnegie Mellon, met it. HARPY handled 1,011 words. It did so by compiling every sentence the task allowed into a single enormous network of about 15,000 states, and walking it with a beam search — keeping not one best path but a small crowd of near-misses, so a temporary wrong turn could still be recovered. It ran at roughly 80 times slower than real time on a machine of about a third of a MIPS, and cost around five dollars a sentence.
The other thing that happened in the 1970s was less popular. Jelinek's group at IBM kept throwing away human knowledge about language and replacing it with counting — and kept getting better results. The famous version of the remark is "Every time I fire a linguist, the performance of our speech recognition system goes up." Jelinek himself later remembered saying something milder. Either way, it is the epitaph of an entire approach, and it stung for decades.
Word error rate on conversational telephone speech (the Switchboard benchmark) and, before it existed, the error rates of the systems of the day on their own much easier tasks. Read the shape, not the individual dots: these are not the same test.
The last stretch is the fastest. In 2012, eleven researchers from four competing labs — Toronto, Microsoft, Google and IBM — did something unusual and published together: Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups. They had all independently found the same thing. Replace the Gaussians in the acoustic model with a deep neural network and the errors drop, everywhere, on every task. In October 2016 Microsoft reported 5.9 percent word error on Switchboard and called it human parity; in August 2017 they reported 5.1 percent against a more carefully measured human benchmark.
And through all of it, the trellis in Act three is still there, under everything. The acoustic model changed out of all recognition. The idea that you must search for the most likely sentence, not the most likely sound, never did.
Six things this page is not sure about
A beautiful wrong picture is worse than a plain right one. Here is where this page is thin, guessing, or telling you a useful lie.
Seventy-five years in thirteen steps
The examination
Try this at the kitchen table
You need: two or more people, a pencil, and a radio or a tap running — anything that makes a steady hiss.
Round one. One person writes down a short sentence and reads it aloud, once, with the hiss going, from the other side of the room. Everyone else writes what they heard. Compare. Count the words that came out wrong — that is a word error rate, measured at your table.
Round two. Same thing, but before reading, the speaker announces the topic: "this sentence is about breakfast." Watch the error count fall. Nothing about the sound changed. You just handed everyone a language model.
Boss level. Hunt for pairs that are genuinely identical in the air — "ice cream / I scream", "grade A / grey day", "a nice man / an ice man", "euthanasia / youth in Asia". Write one on a slip, say it into the hiss, and ask which it was. When somebody insists they can hear the difference, have them say both into a phone's voice-typing and watch the phone flip between them depending on what you said first. That flip is λ.
A last question to take away with you. Every sentence you have ever understood, you understood partly by guessing — filling in from what you already expected, before the sound had finished arriving. So: how much of what you think you heard today did you actually hear?