The journey: why tokenizers exist, what "pair encoding" is, and how a 1994 data-compression trick became the tokenizer of every modern LLM.
An LLM can't eat text — it eats token IDs, integers mapped to text chunks. The tokenizer decides those chunks. And "how do you chop text?" is a false-choice trap:
Word-level. Every word = one token. Simple, but: oov for anything unseen (→ <unk>, information loss), vocab explodes (millions of words × inflections × typos), and it can't generalize — bird and birds share nothing.
Character-level. Tiny vocab, zero OOV, but sequences get long (100 chars = 100 tokens = expensive), and single characters carry almost no meaning. A 1000-char article is 1000 attention steps.
The need: a middle ground. Subword units — common words whole, rare words as pieces. That's exactly what pair encoding delivers.
In 1994 Philip Gage published "A New Algorithm for Data Compression" (C Users Journal). The idea, dead simple:
1. Count every adjacent pair of symbols. 2. Merge the most frequent pair into one new symbol. 3. Repeat until nothing repeats.
That's it. It compresses because frequent patterns collapse into single symbols. Gage was compressing bytes; the exact same loop, run on text, is BPE.
In 2016, Sennrich, Haddow & Birch ("Neural Machine Translation of Rare Words with Subword Units") ran Gage's loop on text and got the open-vocabulary tokenizer:
1. Pre-split corpus on whitespace — merges may never cross a space (subwords stay inside words).
2. Count adjacent token pairs, weighted by how often the containing word appears.
3. Merge the most frequent pair everywhere it occurs.
4. Repeat until the vocabulary hits a target size (the budget).
Result: no more <unk>. Rare words decompose into known pieces (lowest → low + est). Common words become single tokens. The "middle ground" exists.
Everything that followed is a tweak on this loop — same greedy merging, different scoring or pretokenization:
Note the tie-break subtlety: when two pairs tie, the winner is arbitrary (whichever the scan hits first). Different tie-breaks → different vocabularies. BPE merges aren't "correct" — they're consistent. This is why every tokenizer you use differs from every other.
This is the real algorithm, running in your browser on a tiny corpus. Each "merge" is exactly what a trainer does. Watch the greedy loop and the budget math.
low×5, lower×2, newest×6, widest×3 (the classic example). Chips = tokens. Golden = characters, green = merged units._SolidGoldMagikarp budget).No new algorithm — but the pretokenization has to go. Standard BPE pre-splits on whitespace; Chinese (and Japanese) have no spaces, so that step fails. That's precisely why SentencePiece (Kudo & Richardson 2018) and byte-level BPE (GPT-2) drop whitespace pretokenization entirely — SentencePiece treats space as an ordinary character (▁); byte-level BPE works on raw UTF-8 bytes, so any script is just byte pairs. Every Chinese model uses one of these two routes.
Chinese then helps BPE: a hanzi is a dense meaning unit, so once there's enough Chinese data, the greedy loop naturally merges frequent characters into whole phrases — Qwen's tokenizer stores 北京大学 as a single token. The catch is the corpus: a Chinese-light tokenizer (early Llama-2, 32k vocab) splits hanzi into bytes, wasting 2–3× tokens:
| Tokenizer | Chinese efficiency (chars / token) |
|---|---|
| Llama-2 (32k, English-heavy) | 0.48 |
| GPT-3.5 | ~0.65 |
| Llama-3 (128k) | ~0.71 |
| DeepSeek-V3 | ~1.09 |
| Qwen-3 (151k, Chinese-heavy) | ~1.16 |
Same text, Qwen-3 uses ~44% fewer tokens than GPT-3.5. Lesson: for Chinese you don't change the merging — you change where merges are allowed to start. Which is… the same axis SuperBPE moves on, one lesson over.
Vocab = the embedding matrix rows = your memory floor and your compression ceiling. Budget trade-off: bigger vocab = fewer tokens per text (cheaper sequences, better semantic capture) but more embedding parameters and rarer, less-trained tokens. Sizes below are the current standard vocab_size values:
| Model | Vocab size | Tokenizer family |
|---|---|---|
| GPT-2 | 50,257 | byte-level BPE |
| Llama 1 / 2 | 32,000 | SentencePiece BPE |
| Mistral 7B | 32,000 | SentencePiece BPE |
| Yi-1.5 | 64,000 | SentencePiece BPE |
| InternLM2 | 103,168 | SentencePiece BPE |
| DeepSeek-V3 | ~129,000 | byte-level BPE |
| Llama 3 | 128,256 | byte-level BPE |
| Qwen2.5 / Qwen-3 | 151,936 | byte-level BPE |
| GLM-4 | 151,552 | SentencePiece BPE |
| Gemma 2 | 256,128 | SentencePiece |
Trend: 50k (2020) → 128k–256k (2024). Modern multilingual + multilingual-heavy models pay the bigger embedding table because the token savings pay for it. GPT-2's 50k feels cramped today.
Read the original that started it: Sennrich, Haddow & Birch — "Neural Machine Translation of Rare Words with Subword Units" (arXiv 1508.07909). Then Gage's original 1994 compression article for where the loop came from.
Ask your teacher. Confused about any step of the simulator, the tie-breaking, or why Unigram/WordPiece differ from BPE? Ask — I'll walk you through it. Re-run the simulator from scratch in a couple of days and try to predict each merge before clicking: that prediction habit is what converts this from "looks obvious" to "actually yours".