You are the tokenizer. Every sentence in the world stops at your gate, and you chop it into pieces before the machine is allowed to read it.
Take your position. You are a small, stubborn machine standing between every human being who has ever typed anything and the enormous model waiting behind you in the dark. Nothing reaches it except through you. And here is the strange part: you cannot read.
You do not know what an “A” is. You do not know that cat has three letters, or that it is an animal, or that it rhymes with hat. What you have is a numbered list of chunks — pieces of text you have seen so often that you gave each one a number — and one job: cut the incoming sentence into those chunks, as few as possible, and send the numbers through.
The model behind you never sees your scissors. It never sees the letters. It sees a line of numbers, and it has to build everything it knows — grammar, jokes, arithmetic, the whole of human writing — out of the pieces you happened to hand it. Almost every strange thing an AI does traces back to the shape of your cuts.
This is a real sentence, already through the gate. Press the button to see it the way the model first meets it.
Notice what is missing: there are no words here. “Strawberry” is not one thing. The space before a word is usually glued to the front of it. And nothing — nothing at all — is a letter.
The method you use was not invented for language at all. In February 1994 a programmer named Philip Gage published two pages in The C Users Journal called “A New Algorithm for Data Compression.” No neural networks. No linguistics. Just a way to make files smaller.
His idea is almost childishly simple. Look through the data for the pair of neighbouring bytes that occurs most often. Replace every one of them with a single byte that isn’t being used. Write down what you did in a little table. Then do it again. And again. Each pass makes the data shorter and the table longer.
Run it on the string aaabdaaabac — Gage’s own example — and watch.
Press Find the commonest pair, then Merge it. Repeat until nothing repeats. Every merge is a real merge — the string below is being rewritten in front of you.
Gage’s version had a hard ceiling: it could only invent new symbols out of byte values nobody was using, and there are only 256 bytes in the world. It was a neat, modest piece of engineering, and for twenty-one years almost nobody in language research looked at it.
By 2015 machines were translating with neural networks, and they had a humiliating weakness. A translation model carried a fixed word list — typically 30,000 to 50,000 words. Any word outside it came out as a black box: UNK. Unknown.
Which is fatal, because language does not have a fixed word list. German glues nouns together as long as it likes. Names, places, chemicals, misspellings, brand-new slang — all unknown. As Rico Sennrich, Barry Haddow and Alexandra Birch put it at the University of Edinburgh: translation is an open-vocabulary problem.
“We adapt byte pair encoding (BPE) (Gage, 1994), a compression algorithm, to the task of word segmentation.”Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units, arXiv:1508.07909 (v1, 31 August 2015); ACL 2016, Berlin, pp. 1715–1725
That is the whole invention. Take the twenty-one-year-old file-compression trick, point it at text, and stop before it goes too far. Now no word is ever unknown, because in the worst case a word is spelled out of its own letters.
Throw a word at the wall. On the left is a model with a fixed word list. On the right is the same model that has learned pieces instead of words.
The pieces on the right come from the tokenizer you are about to train yourself, two sections down. Nothing here is faked — change its settings later and come back, and this wall changes too.
This is the centre of the lesson, and none of it is pre-cooked. Below is a passage of text. Your browser will read it, count every pair of neighbouring characters in it, merge the commonest pair, and repeat — exactly the way a real tokenizer is built. The only difference is scale: real ones read hundreds of billions of characters and do this fifty thousand times.
Drag the merge slider slowly and watch the test sentence. At zero merges every single character is its own token — the most expensive way to write anything. As you merge, letters fuse into pairs, pairs into fragments, fragments into whole common words. The machine is not learning what words mean. It is only learning what sticks together.
Rank 1 was the commonest pair in the whole passage. Read down the list and you are reading the statistics of a language, discovered by counting alone.
Characters carried per token, measured on the passage
The curve starts a shade below 1 because the end-of-word marker (·) is counted as a token too.
Here is the most famous small humiliation in modern AI: ask a very large, very expensive model how many letter r’s are in the word strawberry, and for a long time it would confidently answer two.
People took this as proof the machine was stupid. It is better read as proof that the machine is blindfolded — and you are the blindfold.
Below is the word as the model receives it. Try to answer the question using only what you can see.
How many letter r’s are in there?
This is why the trick that actually works is to make the model write the word out one letter at a time first. Spaced out, every letter becomes its own token — and suddenly there is something in the numbers to count. The knowledge was never missing. The resolution was.
A tokenizer learns its chunks from whatever it was fed. Feed it mostly English and it will learn long, efficient English chunks — and nothing for the rest of the world. Those languages then arrive as a hail of tiny pieces.
That is not a cosmetic problem. Models are billed by the token, they think in a fixed window of tokens, and they answer more slowly when there are more of them. In 2023 Aleksandar Petrov and colleagues at Oxford measured this across the tokenizers in common use and found the same text could come out up to fifteen times longer depending on the language it was written in.
One sentence, the same meaning, five languages — put through the tokenizer trained on the English passage. Characters that never appeared in English fall back to raw bytes, exactly as a real byte-level tokenizer does.
English-trained tokenizer, 400 merges
| Language | The sentence | Characters | Tokens | × English |
|---|
Honest note: these are this page’s own numbers from a tiny tokenizer trained on one paragraph, not GPT-4’s. The direction of the effect is real and reproducible; the exact multiples are not the published ones.
Here is a subtlety that bit the whole field. The tokenizer is trained on one pile of text. The model is then trained on a different pile — usually a cleaner one. Any chunk that was common in the first pile but got scrubbed out of the second gets a number in the vocabulary and then… nothing. No examples. Its little patch of the model’s memory is never corrected from its random starting value.
On 5 February 2023 Jessica Rumbelow and Matthew Watkins published a list of about 141 such tokens in GPT-2 and GPT-3. The most famous is SolidGoldMagikarp. Asked to repeat it, the model would refuse, or say a different word entirely, or become weirdly hostile. Another researcher traced the source: r/counting, a corner of Reddit where people take turns counting upward forever. A handful of usernames appeared tens of thousands of times in nothing but number posts — common enough to earn their own token, and then filtered out of the training data as junk.
Every cell is a token. Brightness is how much the model actually saw it during training. The dark, flickering ones exist — but were never taught. Click one.
Nothing selected. Find a dark cell.
The Scissors Game. Take any page of a book. With a pencil, go through and find the two letters that appear side by side most often — in English it will almost certainly be th or he or in. Circle every occurrence and give the pair a single new symbol, say a triangle. Now rewrite one paragraph using your symbol. Count how many pencil strokes you saved. Then do it again on your shortened text. Five rounds in, your paragraph will be full of strange little glyphs that stand for the, ing, and — and you will have built, by hand, the same thing that sits in front of every large language model on Earth.
Then the boss level: hand your symbol table to somebody else, along with a word you never used — their name, say — and ask them to write it in your symbols. Watch what happens. That is the open-vocabulary problem, discovered at your own table.
And one question to carry away. If a machine can only ever think in the pieces we hand it — and we chose those pieces by counting, with no idea what any of them meant — how much of what it knows was decided by a pair of scissors?