Tokenization is the step where raw text gets converted into the discrete units — tokens — that a language model actually processes. A model never reads letters or whole words directly; every input is first split into a sequence of tokens drawn from a fixed vocabulary, each mapped to an integer ID, before anything resembling understanding happens. Nearly every downstream property of a model — context length limits, multilingual performance, even how it handles rare words — traces back to how its tokenizer was built.
What changed in 2026
- Multilingual tokenizer quality became a bigger competitive differentiator. Models now compete meaningfully on how efficiently their tokenizer handles non-English scripts, since a poorly-tuned vocabulary can silently double or triple the token cost of the same text in another language.
- Tokenizer-free and byte-level approaches gained more serious research traction, aiming to remove tokenization as a separate step entirely, though byte-pair-based subword tokenization remains dominant in production models.
- Vocabulary sizes crept upward in several frontier releases, trading a larger embedding table for shorter average sequence lengths and better multilingual coverage.
Why subword tokenization exists
Two naive options bookend the design space: tokenize by whole word, or tokenize by individual character. Whole-word tokenization produces a vocabulary that cannot handle words it has never seen (a real problem for names, typos, and rare technical terms) and requires an enormous vocabulary to cover a language well. Character-level tokenization handles any input but produces very long sequences, which is expensive given that attention cost grows quadratically with sequence length (see attention mechanism explained). Subword tokenization is the practical middle ground: common words stay as single tokens, while rare or unfamiliar words get split into smaller, still-meaningful pieces.
How byte-pair encoding builds a vocabulary
- Start with individual characters (or bytes) as the initial vocabulary.
- Count the most frequently occurring adjacent pairs across a large training corpus.
- Merge the most frequent pair into a new single token, and add it to the vocabulary.
- Repeat the counting and merging process thousands of times, until the vocabulary reaches its target size (commonly in the tens of thousands to low hundreds of thousands of tokens).
The result is a vocabulary where common words and word fragments ("the," "ing," "tion") get their own single token, while unusual words get built from multiple smaller pieces. This is why an uncommon technical term or a name in a language the tokenizer was not tuned for often costs several tokens instead of one.
Tokenizer design tradeoffs
| Choice |
Effect of going bigger |
Effect of going smaller |
| Vocabulary size |
Shorter sequences, larger embedding table |
Longer sequences, smaller embedding table |
| Merge count (BPE steps) |
More whole-word tokens, better common-case efficiency |
More character-level fallback for rare words |
| Multilingual coverage |
Better non-English efficiency, larger vocabulary needed |
Cheaper vocabulary, worse non-English efficiency |
| Byte-level fallback |
Handles any input, including unseen scripts and emoji |
N/A — usually included as a safety net regardless |
Why token counts vary between models
Because each model trains its own tokenizer on its own corpus, the same sentence can produce a different number of tokens in different models. A tokenizer trained mostly on English text will tend to represent English efficiently and non-English languages less efficiently, splitting more words into smaller pieces. This directly affects real-world costs and context budgets: two models with identical "128K token" context windows can fit meaningfully different amounts of actual text if their tokenizers differ in efficiency, particularly for non-English content or code.
FAQ
Is a token the same as a word?
No. Common short words are often a single token, but longer or rarer words frequently split into two or more subword tokens. As a rough estimate for English text, one token is roughly three-quarters of a word, but this varies by content.
Why do different AI models charge different amounts for the same text?
Partly because their tokenizers differ — the same text can produce a different token count across models, and pricing is usually per token, so tokenizer efficiency directly affects real cost.
Does a bigger vocabulary always make a model better?
Not automatically. It shortens sequences and can improve multilingual handling, but it also enlarges the embedding table and can leave rarer tokens under-trained if the vocabulary is too large relative to the training data.
What happens when a tokenizer encounters a character it has never seen?
Well-designed tokenizers include a byte-level fallback, so even completely unfamiliar characters or scripts can still be represented, just at the cost of using more tokens than a well-covered language would need.
Where to go next