{"post":{"slug":"tokenizer-training-explained-2026","title":"Tokenizer Training: The Decision Made Before the Model","excerpt":"A tokenizer is trained before the model and cannot be changed afterwards. Its choices determine how efficiently every language and domain is represented, permanently.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">B</span>efore any model weights exist, someone decides how text becomes tokens. That decision is made by training a tokenizer on a corpus, it produces a fixed vocabulary, and it cannot be revised afterwards — every embedding the model learns is tied to that specific token set.\n\nThe consequences are permanent and unevenly distributed.\n\n## What changed in 2026\n\n- **Multilingual efficiency got measured.** The token-count disparity across languages became a documented fairness and cost issue rather than an implementation detail.\n- **Larger vocabularies became common.** Bigger vocabularies improved efficiency for under-represented languages at the cost of a larger embedding layer — see [vocabulary size](/blog/vocabulary-size-explained-2026).\n- **Number handling improved.** Deliberate digit-level treatment addressed long-standing arithmetic weaknesses.\n- **Tokenizers diverged between model families.** Cross-model token count comparisons became unreliable, which matters for cost estimation.\n\n## How it is trained\n\nThe common approach starts from individual characters and repeatedly merges the most frequent adjacent pair, building up subword units until the vocabulary reaches a target size.\n\nThe result is that **frequent sequences in the training corpus become single tokens** and rare ones stay fragmented. That is efficient by construction for whatever the corpus contained, and it is precisely why corpus composition matters so much.\n\nIf English dominates the tokenizer training data, common English words become single tokens while words in less-represented languages fragment into several. The same meaning costs more tokens, which means more money, more context consumed, and slower generation — permanently, for every user of that language.\n\n| Content type | Typical efficiency | Reason |\n| --- | --- | --- |\n| Dominant training language | High | Common words are single tokens |\n| Under-represented languages | Low | Fragmented into pieces |\n| Non-Latin scripts | Often lowest | Frequently byte-level fallback |\n| Code | Moderate | Depends on code in the corpus |\n| Numbers | Historically poor | Inconsistent digit grouping |\n\n## Why numbers and code are awkward\n\n**Numbers** get merged based on frequency, so common sequences become single tokens while uncommon ones split arbitrarily. The same digit can be tokenised differently depending on its neighbours, which makes positional reasoning harder than it should be. This contributed to persistent arithmetic weaknesses, and deliberate digit-level tokenisation has been the fix.\n\n**Code** depends entirely on how much code was in the tokenizer corpus. Where it was well-represented, common constructs and indentation patterns become efficient single tokens. Where it was not, code fragments badly — and since code is verbose, that inefficiency compounds.\n\nWhitespace handling matters particularly for code. A tokenizer that treats runs of spaces as single tokens handles indented code far more efficiently than one that does not, and the difference is large on real files.\n\n## Why it cannot change\n\nThe model's embedding layer maps token IDs to vectors. Those vectors are learned during training, and each one is meaningless outside the tokenizer that produced its ID.\n\nChange the tokenizer and every ID means something different. The embeddings become nonsense, and the model is effectively untrained.\n\nSo a tokenizer decision made at the start of pretraining is locked for the model's life. Extending a vocabulary is possible — adding tokens with newly-initialised embeddings and training them — and it is a substantial intervention rather than a configuration change.\n\nFor anyone not pretraining, the practical implication is simple: you use the base model's tokenizer, and its efficiency characteristics are a property of the model you chose. If your workload is heavily non-English or code-dense, tokenizer efficiency is a legitimate model selection criterion that rarely appears on a comparison chart.\n\n## Common mistakes\n\n- **Comparing token counts across models.** Different tokenizers, incomparable numbers.\n- **Estimating cost for multilingual traffic using English counts.** Can understate substantially.\n- **Assuming a character-count proxy.** The ratio varies by language and content type.\n- **Training a tokenizer for fine-tuning.** You must use the base model's.\n- **Ignoring tokenizer efficiency in model selection.** A real cost factor for non-English workloads.\n- **Not measuring on your actual content.** Published ratios are for general text.\n\n## FAQ\n\n**How do I know my tokenizer's efficiency for my content?**\nEncode a representative sample and divide by character count. Do this for each language and content type you serve; the variation is frequently larger than expected.\n\n**Can I extend a vocabulary?**\nTechnically yes, by adding tokens and training their embeddings. It is a real training intervention with meaningful cost, not a configuration change, and it is rarely worth it outside specialised domains.\n\n**Does a larger vocabulary always help?**\nIt improves token efficiency and enlarges the embedding and output layers, which costs memory and compute. There is a genuine tradeoff — see [vocabulary size](/blog/vocabulary-size-explained-2026).\n\n**Does this affect model quality or only cost?**\nBoth. Fragmented representation makes patterns harder to learn, so heavily fragmented languages tend to see worse quality as well as higher cost.\n\n## Where to go next\n\nFor the size tradeoff, read [vocabulary size](/blog/vocabulary-size-explained-2026). For how context length interacts with token efficiency, [context extension](/blog/context-extension-explained-2026), and for the cost consequences, [LLM unit economics](/blog/llm-unit-economics-2026).","hue":200,"metaTitle":"Tokenizer Training: The Decision Made Before the Model","metaDescription":"How tokenizer training data determines multilingual efficiency, why code and numbers tokenise poorly, and why the choice is permanent.","keywords":["tokenizer training","byte pair encoding","multilingual tokenization","token efficiency","vocabulary construction","tokenizer fairness","code tokenization","llm fundamentals 2026"],"tldr":["**The tokenizer is fixed once the model trains** — changing it invalidates every learned embedding.","**Training data composition determines efficiency** — languages under-represented in that data cost more tokens forever.","**Code and numbers tokenise badly by default** — unless deliberately accounted for.","**Token count is a cost multiplier per language** — the same sentence can differ severalfold across tokenizers.","**Skip:** training your own tokenizer unless you are pretraining. Fine-tuning uses the base model's."],"readTime":"4 min read","wordCount":838},"related":[{"slug":"tokenizers-explained-2026","title":"Tokenizers Explained: How Text Becomes Numbers","excerpt":"A tokenizer splits text into the units a model actually sees, and its choices explain several odd model behaviours. Here is how it works and why it matters.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":214},{"slug":"what-is-tokenization-in-llms-2026","title":"What Is Tokenization in LLMs in 2026? Full Explainer","excerpt":"Before a language model reads anything, tokenization breaks text into the subword units it actually operates on. Here is how that process works and why it matters.","category":"ai","date":"July 24, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200},{"slug":"vocabulary-size-explained-2026","title":"Vocabulary Size: Trading Memory for Token Efficiency","excerpt":"A larger vocabulary means fewer tokens per document and a much larger embedding layer. The tradeoff has moved as models and context windows have grown.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":198}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"shredding-schedule-guide-2026","title":"Shredding Schedule: What to Destroy and When","excerpt":"Keeping every document forever creates a pile nobody can search and a target if anyone breaks in. Knowing what to keep, for how long, and how to destroy the rest resolves both.","category":"guides","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":163},"older":{"slug":"tree-search-agents-2026","title":"Tree Search Agents: Exploring Alternatives Instead of Committing","excerpt":"A normal agent picks a path and follows it. A search agent explores several, scores them, and backtracks from dead ends — which is more thorough and dramatically more expensive.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":213}}