Byte pair encoding, or BPE, is the algorithm that decides how a large language model chops raw text into the tokens it actually computes over. It was originally a 1994 data-compression trick, repurposed for neural machine translation in 2015, and it is now the tokenization backbone behind most production LLMs. Understanding it matters for practical reasons: token count drives API cost, context window usage, and even how fairly a model treats different languages.
What changed in 2026
- Vocabulary sizes kept growing. Multilingual and code-heavy models pushed toward larger vocabularies — often well over 100,000 tokens — to reduce the number of tokens needed to represent non-English text and source code. Exact sizes vary by model and provider; check current documentation rather than assuming a fixed number.
- Byte-level BPE stayed the practical default for most frontier and open-weight model families, because it sidesteps out-of-vocabulary failures entirely by operating on raw UTF-8 bytes rather than pre-defined characters.
- Tokenizer-free research matured further. Byte- and patch-level architectures that skip explicit subword tokenization continued to appear in research papers through 2025 and 2026, though none has displaced BPE-family tokenizers in mainstream production models yet.
- Tokenizer efficiency became a bigger cost lever. As inference cost and context limits stayed a binding constraint, teams increasingly evaluated tokenizer compression ratio — tokens per character for their actual traffic mix — as part of model selection, not just raw benchmark scores.
How byte pair encoding actually works
BPE trains a merge table from a large text corpus in a few repeatable steps:
- Start with a base vocabulary. For byte-level BPE this is simply the 256 possible byte values; for character-level BPE it is the set of unique characters in the training corpus.
- Count every adjacent pair of tokens across the corpus.
- Merge the single most frequent pair into a new token and add it to the vocabulary.
- Repeat — recount pairs, merge the next most frequent one — until the vocabulary reaches a target size, often tens of thousands of merges.
- Freeze the merge table. At inference time, new text is tokenized by greedily applying the learned merges in the order they were learned, not by recomputing frequencies.
The result is a vocabulary of subword units: common words often become a single token, rare words get split into smaller frequent fragments, and truly novel strings fall back to individual bytes or characters. This is why an LLM can process a made-up word or a typo without crashing — it simply falls back to smaller pieces.
BPE vs other tokenization schemes
| Scheme |
Base unit |
Handles unknown input |
Common use |
| Byte-level BPE |
Raw bytes (256 base tokens) |
Always, by construction |
GPT-family and many open-weight models |
| Character-level BPE |
Unicode characters |
Falls back to characters |
Older or language-specific tokenizers |
| WordPiece |
Characters, likelihood-based merges |
Uses an explicit unknown token in some implementations |
BERT-family encoder models |
| Unigram (SentencePiece) |
Probabilistic subword units, pruned top-down |
Always, via subword fallback |
Many multilingual and instruction-tuned models |
The practical difference between BPE and Unigram is the merge strategy: BPE builds up greedily from frequent pairs, while Unigram starts with a large candidate vocabulary and prunes it down using a probabilistic model. Both are still "subword tokenization" in the everyday sense, and both largely solved the older WordPiece-era unknown-token problem.
Why tokenization choices matter in practice
Token count is not a neutral implementation detail — it is the unit that API pricing, context window limits, and generation speed are all measured in. A tokenizer that represents a given language or domain (dense code, non-Latin scripts, technical notation) in fewer tokens gives that content a real cost and context advantage over a tokenizer that fragments it heavily. This is one reason multilingual fairness in LLMs is partly a tokenization problem, not just a training-data problem — see our guide to AI bias detection tools for how teams evaluate this kind of disparity.
Common pitfalls
- Comparing token counts across models as if they mean the same thing. Two tokenizers can encode the same sentence into a very different number of tokens; do not assume cost or context usage transposes cleanly between providers.
- Assuming byte-level BPE means "no tokenization artifacts." It removes unknown-token failures, but merge boundaries can still split meaningful units awkwardly (numbers, rare technical terms), which affects arithmetic and precise string tasks.
- Ignoring tokenizer mismatch when fine-tuning. Reusing training data prepared for one tokenizer against a model with a different one silently degrades quality; always retokenize with the target model's exact tokenizer.
FAQ
Is BPE the same thing as a "tokenizer"?
Not quite. BPE is the algorithm used to build the merge rules; the tokenizer is the software that applies those learned rules to convert text into token IDs at training and inference time.
Why do some words become one token and others become many?
Frequency during training. Common words and word fragments earned their own token because they appeared often enough in the training corpus to justify a merge; rare or novel words get broken into smaller, more frequent pieces.
Does a bigger vocabulary always mean a better tokenizer?
No. Larger vocabularies reduce tokens-per-character for covered content but increase the embedding table size and can hurt performance on genuinely rare tokens that appear too infrequently during training to be learned well.
Can I inspect how a specific model tokenizes my text?
Most providers publish a tokenizer library or an interactive tokenizer tool. Use the actual tokenizer for the model you are calling — do not estimate token counts from a different model's tokenizer.
Where to go next