Before any model weights exist, someone decides how text becomes tokens. That decision is made by training a tokenizer on a corpus, it produces a fixed vocabulary, and it cannot be revised afterwards — every embedding the model learns is tied to that specific token set.
The consequences are permanent and unevenly distributed.
What changed in 2026
- Multilingual efficiency got measured. The token-count disparity across languages became a documented fairness and cost issue rather than an implementation detail.
- Larger vocabularies became common. Bigger vocabularies improved efficiency for under-represented languages at the cost of a larger embedding layer — see vocabulary size.
- Number handling improved. Deliberate digit-level treatment addressed long-standing arithmetic weaknesses.
- Tokenizers diverged between model families. Cross-model token count comparisons became unreliable, which matters for cost estimation.
How it is trained
The common approach starts from individual characters and repeatedly merges the most frequent adjacent pair, building up subword units until the vocabulary reaches a target size.
The result is that frequent sequences in the training corpus become single tokens and rare ones stay fragmented. That is efficient by construction for whatever the corpus contained, and it is precisely why corpus composition matters so much.
If English dominates the tokenizer training data, common English words become single tokens while words in less-represented languages fragment into several. The same meaning costs more tokens, which means more money, more context consumed, and slower generation — permanently, for every user of that language.
| Content type |
Typical efficiency |
Reason |
| Dominant training language |
High |
Common words are single tokens |
| Under-represented languages |
Low |
Fragmented into pieces |
| Non-Latin scripts |
Often lowest |
Frequently byte-level fallback |
| Code |
Moderate |
Depends on code in the corpus |
| Numbers |
Historically poor |
Inconsistent digit grouping |
Why numbers and code are awkward
Numbers get merged based on frequency, so common sequences become single tokens while uncommon ones split arbitrarily. The same digit can be tokenised differently depending on its neighbours, which makes positional reasoning harder than it should be. This contributed to persistent arithmetic weaknesses, and deliberate digit-level tokenisation has been the fix.
Code depends entirely on how much code was in the tokenizer corpus. Where it was well-represented, common constructs and indentation patterns become efficient single tokens. Where it was not, code fragments badly — and since code is verbose, that inefficiency compounds.
Whitespace handling matters particularly for code. A tokenizer that treats runs of spaces as single tokens handles indented code far more efficiently than one that does not, and the difference is large on real files.
Why it cannot change
The model's embedding layer maps token IDs to vectors. Those vectors are learned during training, and each one is meaningless outside the tokenizer that produced its ID.
Change the tokenizer and every ID means something different. The embeddings become nonsense, and the model is effectively untrained.
So a tokenizer decision made at the start of pretraining is locked for the model's life. Extending a vocabulary is possible — adding tokens with newly-initialised embeddings and training them — and it is a substantial intervention rather than a configuration change.
For anyone not pretraining, the practical implication is simple: you use the base model's tokenizer, and its efficiency characteristics are a property of the model you chose. If your workload is heavily non-English or code-dense, tokenizer efficiency is a legitimate model selection criterion that rarely appears on a comparison chart.
Common mistakes
- Comparing token counts across models. Different tokenizers, incomparable numbers.
- Estimating cost for multilingual traffic using English counts. Can understate substantially.
- Assuming a character-count proxy. The ratio varies by language and content type.
- Training a tokenizer for fine-tuning. You must use the base model's.
- Ignoring tokenizer efficiency in model selection. A real cost factor for non-English workloads.
- Not measuring on your actual content. Published ratios are for general text.
FAQ
How do I know my tokenizer's efficiency for my content?
Encode a representative sample and divide by character count. Do this for each language and content type you serve; the variation is frequently larger than expected.
Can I extend a vocabulary?
Technically yes, by adding tokens and training their embeddings. It is a real training intervention with meaningful cost, not a configuration change, and it is rarely worth it outside specialised domains.
Does a larger vocabulary always help?
It improves token efficiency and enlarges the embedding and output layers, which costs memory and compute. There is a genuine tradeoff — see vocabulary size.
Does this affect model quality or only cost?
Both. Fragmented representation makes patterns harder to learn, so heavily fragmented languages tend to see worse quality as well as higher cost.
Where to go next
For the size tradeoff, read vocabulary size. For how context length interacts with token efficiency, context extension, and for the cost consequences, LLM unit economics.