A language model never sees text. It sees a sequence of integers, each representing a chunk of characters the tokenizer decided was a useful unit. Those chunks are learned from training data rather than defined by linguistics, which is why they do not line up with words, syllables, or anything else you might expect.
Several persistent model quirks stop being mysterious once you understand this layer.
What changed in 2026
- Vocabularies grew. Newer tokenizers used substantially larger vocabularies, improving efficiency on code and non-English text at the cost of a larger embedding layer.
- Multilingual fairness got attention. The cost and context-length penalty for non-Latin scripts became a documented concern rather than an unnoticed artifact.
- Byte-level fallbacks became universal. Handling any input without unknown-token failures, including emoji and unusual scripts, became standard.
- Tokenizer-free research continued. Approaches operating directly on bytes advanced without displacing subword tokenization in production models.
Why the odd behaviours happen
| Behaviour |
Tokenizer explanation |
| Cannot reliably count letters in a word |
It sees a token, not the characters inside it |
| Struggles to reverse a string |
Same reason; character positions are invisible |
| Arithmetic errors on long numbers |
Numbers split into inconsistent chunks |
| Non-English costs more |
Fewer learned patterns, so more fragments per word |
| Rare names spelled oddly |
Split into fragments with no strong association |
| Whitespace changes affect output |
Leading spaces are part of tokens |
| Some prompts behave strangely at boundaries |
Token splits interact with formatting in unintuitive ways |
The letter-counting case is the most cited and the most instructive. Asking how many times a letter appears in a word requires seeing characters, and the model receives a token identifier for the whole word or a chunk of it. It has learned associations that let it often guess correctly, and it is not reading the letters. This is why the reliable fix is giving the model a tool that counts, not a better prompt.
Number handling has the same root. If a long number splits into chunks inconsistently depending on surrounding characters, arithmetic on it is operating over an unstable representation. Again, a calculator tool solves what prompting cannot.
Practical consequences
For cost, tokenizer efficiency directly determines what you pay, and it varies substantially by content type and language — the detail is in token counting explained.
For context, the same window holds less text in languages that tokenize inefficiently. A limit expressed in tokens is not a uniform limit in meaning across locales.
For prompt design, be aware that formatting changes token boundaries. Consistent structure tends to produce more consistent behavior than prose that varies in spacing and punctuation, though this is a minor effect compared to content.
For task design, route character-level and arithmetic work to tools rather than trying to prompt around a representational limit. That is a tool-calling problem, covered in tool calling reliability.
Common mistakes
- Prompting harder at character-level tasks. The information is not available to the model; give it a tool.
- Assuming token counts transfer between models. Different vocabularies, different counts.
- Ignoring multilingual cost differences. Real, measurable, and worth measuring for your locales.
- Embedding base64 or long identifiers. Extremely inefficient; no learned patterns to compress them.
- Treating tokens as words in capacity planning. The ratio varies too much.
FAQ
Why not just use characters as tokens?
Sequences would be much longer, and attention cost grows with sequence length. Subword units are a compromise between vocabulary size and sequence length.
Do all models use the same tokenizer?
No. Each model family has its own, which is why counts and efficiency differ between them.
Can I see how my text tokenizes?
Yes. Most provider SDKs include a tokenizer you can run locally, and several web tools visualize the splits.
Does a bigger vocabulary always help?
It improves efficiency and enlarges the embedding layer, adding parameters. It is a tradeoff that model designers tune rather than a free improvement.
Where to go next
For the cost side, read token counting explained. For reducing prompt size, prompt compression, and for embeddings built on the same units, sparse embeddings explained.