Vocabulary size is one of the earliest architectural decisions and one of the least discussed. It determines how many distinct tokens exist, which determines how finely text is chopped, which determines how much of a document fits in a context window.
It also determines the size of two of the model's largest layers.
What changed in 2026
- Vocabularies grew substantially. Modern models moved well beyond the sizes common a few years earlier.
- Multilingual pressure drove the increase. Better representation for non-dominant languages was a primary motivation.
- Longer contexts changed the calculation. With very large context windows, token efficiency matters differently than when windows were tight.
- Output layer cost got attention. As vocabularies grew, the per-token cost of computing a distribution over all of them became a design consideration.
What grows and what shrinks
|
Smaller vocabulary |
Larger vocabulary |
| Tokens per document |
More |
Fewer |
| Text fitting in context |
Less |
More |
| Embedding layer size |
Smaller |
Larger |
| Output layer size |
Smaller |
Larger |
| Per-token output computation |
Cheaper |
More expensive |
| Rare word handling |
Fragmented |
Better represented |
| Multilingual efficiency |
Worse |
Better |
Two layers scale directly with vocabulary size: the embedding layer, mapping token IDs to vectors, and the output layer, producing a score for every token at every step. Both are vocabulary size times model dimension, and at large vocabularies they become a meaningful share of total parameters.
The output layer is the sharper constraint, because it is computed at every generated token. A distribution over a very large vocabulary is more expensive to produce than over a small one, every single step. That is a recurring inference cost rather than a one-off memory cost.
Why bigger became better
Despite that cost, vocabularies grew, for reasons that make sense once you look at where the inefficiency lands.
Multilingual fairness and efficiency. A small vocabulary trained on a corpus dominated by one language fragments everything else badly. Enlarging it allows common words from more languages to become single tokens, which reduces cost and improves quality for those users — see tokenizer training.
Context efficiency. Fewer tokens per document means more real content in the same window. With retrieval-augmented workloads packing large amounts of context into every request, that is a direct efficiency gain.
Sequence length effects. Attention cost grows with sequence length, so representing the same text in fewer tokens reduces attention computation. That partly offsets the larger output layer.
The net has favoured larger vocabularies as models grew, because the embedding and output layers are a shrinking fraction of a large model's parameters while the token efficiency benefit applies to every request.
What it means if you are not pretraining
For almost everyone, this is a property of the model you chose rather than a decision you make.
The practical consequence is in cost estimation and model selection. Two models with different vocabularies produce different token counts for identical text, so their published per-token prices are not directly comparable. A model with a more efficient tokenizer for your content may be cheaper in practice despite a higher per-token rate.
The way to check is to encode a representative sample of your actual content with each candidate's tokenizer and compare total tokens, then multiply by price. For heavily non-English or code-dense workloads, the difference can reverse an apparent price advantage.
Common mistakes
- Comparing per-token prices without comparing token counts. Different tokenizers, different counts.
- Assuming a fixed characters-per-token ratio. Varies by language, content, and model.
- Ignoring it for multilingual products. The efficiency gap is largest exactly there.
- Treating vocabulary size as a quality indicator. It is a tradeoff, not a score.
- Estimating context capacity in words. Measure in tokens with the actual tokenizer.
FAQ
Does a larger vocabulary make a model better?
It makes it more token-efficient, particularly for under-represented content. Quality depends on far more than vocabulary size, so it is not a ranking criterion.
How does this affect context window capacity?
Directly. A model with a more efficient tokenizer for your content fits more real text into the same nominal window — so two models with identical stated window sizes may hold different amounts of your documents.
Can vocabulary size change after training?
Not without adding new embeddings and training them, which is a substantial intervention. It is fixed in practice.
Should this influence model choice?
For multilingual or code-heavy workloads, yes — measure token counts on your content rather than trusting general ratios.
Where to go next
For how the vocabulary is constructed, read tokenizer training. For how context capacity interacts with it, context extension, and for the cost implications, LLM unit economics.