{"post":{"slug":"vocabulary-size-explained-2026","title":"Vocabulary Size: Trading Memory for Token Efficiency","excerpt":"A larger vocabulary means fewer tokens per document and a much larger embedding layer. The tradeoff has moved as models and context windows have grown.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">V</span>ocabulary size is one of the earliest architectural decisions and one of the least discussed. It determines how many distinct tokens exist, which determines how finely text is chopped, which determines how much of a document fits in a context window.\n\nIt also determines the size of two of the model's largest layers.\n\n## What changed in 2026\n\n- **Vocabularies grew substantially.** Modern models moved well beyond the sizes common a few years earlier.\n- **Multilingual pressure drove the increase.** Better representation for non-dominant languages was a primary motivation.\n- **Longer contexts changed the calculation.** With very large context windows, token efficiency matters differently than when windows were tight.\n- **Output layer cost got attention.** As vocabularies grew, the per-token cost of computing a distribution over all of them became a design consideration.\n\n## What grows and what shrinks\n\n| | Smaller vocabulary | Larger vocabulary |\n| --- | --- | --- |\n| Tokens per document | More | Fewer |\n| Text fitting in context | Less | More |\n| Embedding layer size | Smaller | Larger |\n| Output layer size | Smaller | Larger |\n| Per-token output computation | Cheaper | More expensive |\n| Rare word handling | Fragmented | Better represented |\n| Multilingual efficiency | Worse | Better |\n\nTwo layers scale directly with vocabulary size: the embedding layer, mapping token IDs to vectors, and the output layer, producing a score for every token at every step. Both are vocabulary size times model dimension, and at large vocabularies they become a meaningful share of total parameters.\n\nThe output layer is the sharper constraint, because it is computed *at every generated token*. A distribution over a very large vocabulary is more expensive to produce than over a small one, every single step. That is a recurring inference cost rather than a one-off memory cost.\n\n## Why bigger became better\n\nDespite that cost, vocabularies grew, for reasons that make sense once you look at where the inefficiency lands.\n\n**Multilingual fairness and efficiency.** A small vocabulary trained on a corpus dominated by one language fragments everything else badly. Enlarging it allows common words from more languages to become single tokens, which reduces cost and improves quality for those users — see [tokenizer training](/blog/tokenizer-training-explained-2026).\n\n**Context efficiency.** Fewer tokens per document means more real content in the same window. With retrieval-augmented workloads packing large amounts of context into every request, that is a direct efficiency gain.\n\n**Sequence length effects.** Attention cost grows with sequence length, so representing the same text in fewer tokens reduces attention computation. That partly offsets the larger output layer.\n\nThe net has favoured larger vocabularies as models grew, because the embedding and output layers are a shrinking fraction of a large model's parameters while the token efficiency benefit applies to every request.\n\n## What it means if you are not pretraining\n\nFor almost everyone, this is a property of the model you chose rather than a decision you make.\n\nThe practical consequence is in **cost estimation and model selection**. Two models with different vocabularies produce different token counts for identical text, so their published per-token prices are not directly comparable. A model with a more efficient tokenizer for your content may be cheaper in practice despite a higher per-token rate.\n\nThe way to check is to encode a representative sample of your actual content with each candidate's tokenizer and compare total tokens, then multiply by price. For heavily non-English or code-dense workloads, the difference can reverse an apparent price advantage.\n\n## Common mistakes\n\n- **Comparing per-token prices without comparing token counts.** Different tokenizers, different counts.\n- **Assuming a fixed characters-per-token ratio.** Varies by language, content, and model.\n- **Ignoring it for multilingual products.** The efficiency gap is largest exactly there.\n- **Treating vocabulary size as a quality indicator.** It is a tradeoff, not a score.\n- **Estimating context capacity in words.** Measure in tokens with the actual tokenizer.\n\n## FAQ\n\n**Does a larger vocabulary make a model better?**\nIt makes it more token-efficient, particularly for under-represented content. Quality depends on far more than vocabulary size, so it is not a ranking criterion.\n\n**How does this affect context window capacity?**\nDirectly. A model with a more efficient tokenizer for your content fits more real text into the same nominal window — so two models with identical stated window sizes may hold different amounts of your documents.\n\n**Can vocabulary size change after training?**\nNot without adding new embeddings and training them, which is a substantial intervention. It is fixed in practice.\n\n**Should this influence model choice?**\nFor multilingual or code-heavy workloads, yes — measure token counts on your content rather than trusting general ratios.\n\n## Where to go next\n\nFor how the vocabulary is constructed, read [tokenizer training](/blog/tokenizer-training-explained-2026). For how context capacity interacts with it, [context extension](/blog/context-extension-explained-2026), and for the cost implications, [LLM unit economics](/blog/llm-unit-economics-2026).","hue":198,"metaTitle":"Vocabulary Size: Memory Against Token Efficiency","metaDescription":"Why larger vocabularies reduce token counts, what they cost in embedding and output layer size, and how the tradeoff shifted with longer contexts.","keywords":["vocabulary size llm","embedding layer size","token efficiency","softmax cost","multilingual vocabulary","model architecture tradeoffs","tokenizer design","llm fundamentals 2026"],"tldr":["**Larger vocabularies mean fewer tokens per document** — more text fits in the same context window.","**The cost is in the embedding and output layers** — both scale directly with vocabulary size.","**The output layer computation is the real constraint** — a softmax over a very large vocabulary is expensive per token.","**Bigger vocabularies help under-represented languages most** — they suffer worst from fragmentation.","**Skip:** worrying about this unless you are pretraining. It is a property of the model you selected."],"readTime":"4 min read","wordCount":813},"related":[{"slug":"tokenizers-explained-2026","title":"Tokenizers Explained: How Text Becomes Numbers","excerpt":"A tokenizer splits text into the units a model actually sees, and its choices explain several odd model behaviours. Here is how it works and why it matters.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":214},{"slug":"tokenizer-training-explained-2026","title":"Tokenizer Training: The Decision Made Before the Model","excerpt":"A tokenizer is trained before the model and cannot be changed afterwards. Its choices determine how efficiently every language and domain is represented, permanently.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200},{"slug":"what-is-tokenization-in-llms-2026","title":"What Is Tokenization in LLMs in 2026? Full Explainer","excerpt":"Before a language model reads anything, tokenization breaks text into the subword units it actually operates on. Here is how that process works and why it matters.","category":"ai","date":"July 24, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"uuid-v7-explained-2026","title":"UUID v7: Opaque Identifiers That Insert Sequentially","excerpt":"Random UUIDs fragment your storage. Version 7 puts a timestamp in the high bits, so identifiers sort roughly by creation time and insert cleanly at the end.","category":"code","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":304},"older":null}