The rule of thumb is that a token is about three-quarters of a word. That is roughly right for ordinary English prose and wrong for almost everything else you will actually send. Code, JSON, tables, non-English text, and unusual names all tokenize far less efficiently, and budgets built on the rule of thumb come in well under actual spend.
Counting properly takes one library call and removes an entire category of surprise.
What changed in 2026
- Tokenizer efficiency gaps got quantified. Comparative analysis made clear how much more expensive non-English text is across model families, which raised fairness concerns in multilingual products.
- Hidden token categories became visible. Providers documented reasoning token billing and tool definition overhead more clearly after users noticed the gap between estimates and invoices.
- Vocabulary sizes grew. Newer tokenizers with larger vocabularies improved efficiency on code and non-English text relative to earlier generations.
- Counting utilities shipped with SDKs. Accurate local counting became a standard SDK feature rather than a separate dependency.
What tokenizes badly
| Content type |
Tokens per word, roughly |
| Ordinary English prose |
Baseline |
| Technical English with jargon |
Slightly above baseline |
| Code |
Well above; punctuation and indentation cost |
| JSON with long key names |
Well above; braces, quotes, keys repeat |
| Tables and markdown formatting |
Above; delimiters everywhere |
| Common European languages |
Above baseline |
| Languages with non-Latin scripts |
Substantially above |
| Base64 or random identifiers |
Very high; no learned patterns |
| Repeated boilerplate |
Baseline, but repeated every request |
The non-Latin script row has a real consequence: the same question costs materially more in some languages than in English, and the same context window holds less. For a multilingual product this is both a cost issue and a capability difference between locales that is worth measuring rather than assuming away.
Base64 and random identifiers are the pathological case. A tokenizer learns patterns from text, and random strings have none, so they fragment into many tokens. Embedding a base64 image or a long list of identifiers in a prompt is far more expensive than its character count suggests.
The hidden line items
Tool definitions. Every tool schema is sent with every request that could use it. A dozen tools with detailed descriptions is a standing per-request cost that never appears in your prompt text — one more reason the tool-count guidance in tool calling reliability matters financially as well as for accuracy.
Reasoning tokens. Models that think before answering generate tokens you frequently never see, and most providers bill them as output. On a reasoning-heavy workload this can dominate, as covered in test-time compute explained.
System prompts. Counted on every request. A long system prompt is a per-call tax, which is what makes prompt caching worthwhile.
Message history. In a conversation, the whole history is resent each turn. Cost per turn grows through the conversation even when each new message is short.
To count accurately, use the tokenizer for the specific model. Counts differ between model families for identical text, so a count from one is not transferable to another.
Common mistakes
- Estimating with a fixed words-to-tokens ratio. Wrong for code, structured data, and non-English.
- Ignoring tool definition overhead. A standing cost on every request.
- Forgetting reasoning tokens. Frequently the largest line and the least visible.
- Counting only the new message in a chat. History is resent each turn.
- Using one model's tokenizer for another. Counts are not portable.
- Sending base64 in prompts. Extremely token-expensive; use a URL or a file API.
FAQ
How do I count tokens accurately?
Use the tokenizer library for your specific model family, available in most provider SDKs. It is exact rather than an estimate.
Why does the same prompt cost differently on two models?
Different tokenizers segment text differently, so identical input produces different counts.
Are input and output tokens priced the same?
Almost never. Output is typically several times more expensive, which is why response length matters more than prompt length for cost.
Does whitespace count?
Yes. Indentation and blank lines consume tokens, which is part of why formatted code is expensive.
Where to go next
For the mechanism, read tokenizers explained. For reducing what you send, context compression and prompt compression.