{"post":{"slug":"token-counting-explained-2026","title":"Token Counting Explained: Why Your Estimate Is Wrong","excerpt":"Tokens are not words and the ratio varies by language, format, and content. Here is how to count accurately and where hidden tokens inflate your bill.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>he rule of thumb is that a token is about three-quarters of a word. That is roughly right for ordinary English prose and wrong for almost everything else you will actually send. Code, JSON, tables, non-English text, and unusual names all tokenize far less efficiently, and budgets built on the rule of thumb come in well under actual spend.\n\nCounting properly takes one library call and removes an entire category of surprise.\n\n## What changed in 2026\n\n- **Tokenizer efficiency gaps got quantified.** Comparative analysis made clear how much more expensive non-English text is across model families, which raised fairness concerns in multilingual products.\n- **Hidden token categories became visible.** Providers documented reasoning token billing and tool definition overhead more clearly after users noticed the gap between estimates and invoices.\n- **Vocabulary sizes grew.** Newer tokenizers with larger vocabularies improved efficiency on code and non-English text relative to earlier generations.\n- **Counting utilities shipped with SDKs.** Accurate local counting became a standard SDK feature rather than a separate dependency.\n\n## What tokenizes badly\n\n| Content type | Tokens per word, roughly |\n| --- | --- |\n| Ordinary English prose | Baseline |\n| Technical English with jargon | Slightly above baseline |\n| Code | Well above; punctuation and indentation cost |\n| JSON with long key names | Well above; braces, quotes, keys repeat |\n| Tables and markdown formatting | Above; delimiters everywhere |\n| Common European languages | Above baseline |\n| Languages with non-Latin scripts | Substantially above |\n| Base64 or random identifiers | Very high; no learned patterns |\n| Repeated boilerplate | Baseline, but repeated every request |\n\nThe non-Latin script row has a real consequence: the same question costs materially more in some languages than in English, and the same context window holds less. For a multilingual product this is both a cost issue and a capability difference between locales that is worth measuring rather than assuming away.\n\nBase64 and random identifiers are the pathological case. A tokenizer learns patterns from text, and random strings have none, so they fragment into many tokens. Embedding a base64 image or a long list of identifiers in a prompt is far more expensive than its character count suggests.\n\n## The hidden line items\n\n**Tool definitions.** Every tool schema is sent with every request that could use it. A dozen tools with detailed descriptions is a standing per-request cost that never appears in your prompt text — one more reason the tool-count guidance in [tool calling reliability](/blog/tool-calling-reliability-2026) matters financially as well as for accuracy.\n\n**Reasoning tokens.** Models that think before answering generate tokens you frequently never see, and most providers bill them as output. On a reasoning-heavy workload this can dominate, as covered in [test-time compute explained](/blog/test-time-compute-explained-2026).\n\n**System prompts.** Counted on every request. A long system prompt is a per-call tax, which is what makes prompt caching worthwhile.\n\n**Message history.** In a conversation, the whole history is resent each turn. Cost per turn grows through the conversation even when each new message is short.\n\nTo count accurately, use the tokenizer for the specific model. Counts differ between model families for identical text, so a count from one is not transferable to another.\n\n## Common mistakes\n\n- **Estimating with a fixed words-to-tokens ratio.** Wrong for code, structured data, and non-English.\n- **Ignoring tool definition overhead.** A standing cost on every request.\n- **Forgetting reasoning tokens.** Frequently the largest line and the least visible.\n- **Counting only the new message in a chat.** History is resent each turn.\n- **Using one model's tokenizer for another.** Counts are not portable.\n- **Sending base64 in prompts.** Extremely token-expensive; use a URL or a file API.\n\n## FAQ\n\n**How do I count tokens accurately?**\nUse the tokenizer library for your specific model family, available in most provider SDKs. It is exact rather than an estimate.\n\n**Why does the same prompt cost differently on two models?**\nDifferent tokenizers segment text differently, so identical input produces different counts.\n\n**Are input and output tokens priced the same?**\nAlmost never. Output is typically several times more expensive, which is why response length matters more than prompt length for cost.\n\n**Does whitespace count?**\nYes. Indentation and blank lines consume tokens, which is part of why formatted code is expensive.\n\n## Where to go next\n\nFor the mechanism, read [tokenizers explained](/blog/tokenizers-explained-2026). For reducing what you send, [context compression](/blog/context-compression-2026) and [prompt compression](/blog/prompt-compression-2026).","hue":199,"metaTitle":"Token Counting Explained: Accurate Estimates 2026","metaDescription":"How tokenization affects cost, why non-English and code cost more per word, the hidden tokens in tool definitions and reasoning, and how to count accurately.","keywords":["token counting explained","llm token estimation","tokenizer differences","non english token cost","hidden token charges","reasoning tokens billing","context window budgeting","api cost estimation","llm pricing 2026"],"tldr":["**Token counts vary by tokenizer, so counts are model-specific** — the same text costs differently across model families.","**Non-English text costs substantially more per word** — sometimes several times more for the same meaning.","**Code, JSON, and structured text tokenize inefficiently** — punctuation and indentation each consume tokens.","**Reasoning tokens and tool definitions are the hidden line items** — billed and easy to overlook.","**Skip:** estimating with a words-times-a-constant rule for budgeting. Use the actual tokenizer for anything you plan around."],"readTime":"3 min read","wordCount":737},"related":[{"slug":"token-pricing-explained-2026","title":"Token Pricing Explained 2026: How AI Billing Really Works","excerpt":"AI APIs bill by the token, not the word or the request, and input and output tokens are priced differently. Here's what a token actually is and how to estimate a real bill before you get surprised by one.","category":"ai","date":"August 10, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":216},{"slug":"ai-token-limits-explained-2026","title":"AI Token Limits Explained in 2026: Context Windows and Costs","excerpt":"Token limits define how much text a model can read and write at once. This 2026 guide explains what tokens are, how context windows work, and how to manage both.","category":"ai","date":"July 5, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":305},{"slug":"ai-token-pricing-explained-2026","title":"AI token pricing explained for 2026","excerpt":"LLM APIs charge by the token, not the request, and the pricing structure has more moving parts than a single per-token rate. Here is how to read it.","category":"ai","date":"July 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":320}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"streaming-responses-llm-2026","title":"Streaming LLM Responses: Perceived Speed Is the Product","excerpt":"Streaming does not make generation faster, it makes waiting tolerable. Here is how to implement it well, what breaks with structured output, and how to handle failure mid-stream.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":209},"older":{"slug":"tokenizers-explained-2026","title":"Tokenizers Explained: How Text Becomes Numbers","excerpt":"A tokenizer splits text into the units a model actually sees, and its choices explain several odd model behaviours. Here is how it works and why it matters.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":214}}