A token is the basic unit a language model reads and writes in — usually a word, part of a word, or a punctuation mark, depending on the tokenizer. The context window is the maximum number of tokens a model can consider at once, covering the input prompt, any retrieved content, and the output combined. Understanding both is necessary for reasoning about cost, latency, and what a model can actually do in a single call.
What changed in 2026
- Context windows kept expanding across major providers, with some models supporting context lengths that would have been considered exceptional a couple of years earlier — though actual usable context (where the model reliably attends to everything) grows more slowly than the advertised limit.
- Long-context retrieval quality became a benchmark category of its own, since a model technically accepting a huge context does not mean it reasons equally well across all of it — "needle in a haystack" performance now gets reported alongside raw context size.
- Token-efficient context strategies matured, including better chunking and retrieval-augmented approaches that avoid stuffing full documents into every call.
Tokens vs words
Roughly, one token is about three-quarters of an English word on average, though this varies by language, formatting, and content type — code and non-English text often tokenize less efficiently than plain English prose. Always verify actual token counts with the provider's tokenizer or counting tool rather than estimating from word count, especially for cost-sensitive applications.
Context window sizes compared
| Use case |
Typical context need |
Notes |
| Short chat turn |
A few hundred to a few thousand tokens |
Conversation history accumulates fast in multi-turn chats |
| Document summarization |
A few thousand to tens of thousands of tokens |
Depends entirely on document length |
| RAG-based Q&A |
Retrieved chunks plus the question, usually modest |
Retrieval quality matters more than raw window size |
| Codebase-aware assistance |
Can require very large context |
Even large windows may need chunking for big repos |
| Long-form analysis of multiple documents |
Large context, or a retrieval strategy instead |
Bigger is not automatically better here |
Managing token limits in practice
Trim context to what is actually relevant rather than including everything available — irrelevant content in the context window both costs money and can measurably degrade output quality by diluting what the model attends to. For multi-turn conversations, actively manage history: summarize or drop older turns rather than letting the transcript grow unbounded until it hits the limit and truncates unpredictably. For document-heavy tasks, retrieval-based approaches that pull only relevant chunks usually outperform stuffing full documents into context, both on cost and on answer quality.
Cost implications
Token cost scales with both input and output, and for many production applications the input side — system prompts, retrieved context, conversation history — dominates spend far more than the output does. A streaming response does not change token cost; it changes only how the same tokens are delivered. Track token usage per request type, not just in aggregate, to find which call patterns are actually driving spend.
FAQ
What happens if I exceed the context window?
Behavior varies by provider — some return an error, others silently truncate the oldest content. Never assume graceful degradation; test and handle the limit explicitly in your application.
Does a bigger context window always produce better answers?
No. Larger windows increase what a model can technically consider, but retrieval quality across very long context can degrade, and irrelevant content can dilute the response. Relevant, well-chosen context usually beats maximal context.
How do I count tokens before sending a request?
Use the provider's official tokenizer or counting library rather than estimating from word count — the estimate can be meaningfully off, especially for non-English text or code.
Is the context window shared between input and output?
Typically yes — the context window is a combined budget for prompt plus output, not two separate limits. Check the specific model's documentation, since this varies.
Where to go next