A context window is the maximum amount of text, measured in tokens, that a model can hold and actively reason over in a single request, including the prompt, any retrieved documents, and its own output so far. Once a conversation or document exceeds that limit, the oldest content gets truncated or summarized, and the model can no longer see it. Vendors have spent the last two years racing to publish bigger context numbers, and by mid-2026 windows of one million tokens or more are common in flagship models. The number on the spec sheet is real, but it measures capacity, not comprehension, and the gap between the two is where most practical disappointment with long-context features comes from.
How it works
A transformer model processes every token in its context window against every other token through attention, which is why context is not free: doubling the window roughly quadruples the raw attention computation unless the vendor uses a cheaper approximation. That cost pressure is exactly why most providers built their long windows using tricks like sparse or windowed attention rather than brute-force scaling, and why using more of a window inflates the latency and price of a request even when the underlying task has not changed.
Two numbers matter more than the advertised maximum. Advertised context is the figure on the spec sheet: what the model can technically accept as input. Effective context is how much of that the model actually uses correctly at inference time, and it is almost always smaller. Independent long-context evaluations consistently find that models recall facts placed at the very start or end of a long input far more reliably than facts buried in the middle, a pattern generally called the "lost in the middle" effect. A model can accept 500,000 tokens and still answer worse on a fact planted at position 250,000 than on the identical fact planted at position 10.
Marketed context vs what you can actually rely on
| Window size |
Typical marketed use |
What usually still works well |
Where recall gets shaky |
| 8K-32K tokens |
Chat, short documents |
Full recall across the whole window |
Rarely an issue at this size |
| 100K-200K tokens |
A book chapter, a codebase slice |
Facts near the start and end |
Specific details buried in the middle |
| 500K-1M+ tokens |
Whole books, large repos, long transcripts |
Broad summarization, thematic questions |
Precise lookups anywhere but the edges |
The practical rule of thumb: the larger the window, the more you should treat it as a summarization and broad-context tool rather than a precise-retrieval tool, unless you have specifically tested recall at the positions that matter for your task.
When long context beats retrieval, and when it does not
Long context wins when a task genuinely needs the model to reason across scattered pieces of a document at once, such as tracing a variable through an entire codebase or comparing clauses spread across a long contract. Retrieval-augmented generation still wins when the underlying corpus is far bigger than any context window could hold, when a response needs to cite a specific source, or when cost matters, since a retrieval call plus a short prompt is typically far cheaper than resending hundreds of thousands of tokens on every turn. In most production systems the two approaches end up combined rather than treated as a strict either-or choice.
Common mistakes
- Assuming a bigger window means better answers. Larger windows increase capacity, not accuracy; without testing recall at realistic positions, teams routinely overestimate what a model will actually retain.
- Resending the entire conversation history on every turn. This inflates cost and latency for little benefit once a conversation exceeds what actually gets used well; summarizing or trimming older turns is usually cheaper and just as effective.
- Skipping needle-in-haystack testing for the specific use case. Published benchmarks describe general trends; a given organizations documents, formatting, and query style can behave differently and deserve a quick internal test before launch.
- Treating context window size as a proxy for model quality. A smaller, better-trained model can outperform a larger-context model on reasoning tasks; window size is one spec among many, not a ranking signal.
FAQ
Does a 1 million token context window mean I can paste in a million tokens and get a perfect answer?
No. The model will accept that much input, but recall accuracy typically degrades for details in the middle of very long inputs, so treat large windows as good for broad synthesis rather than guaranteed precise retrieval everywhere.
Why do longer prompts cost so much more?
Most providers charge per token, and processing cost scales with how many tokens the model has to attend over, so a 200,000 token prompt is meaningfully more expensive and slower than a 2,000 token prompt even on the same model.
Should I fine-tune a model instead of relying on a huge context window?
They solve different problems. Fine-tuning changes how a model behaves or what style it uses; a context window supplies facts and content the model did not otherwise have. See our guide to parameter-efficient fine-tuning for what fine-tuning actually changes.
Do open models offer competitive context windows compared with closed models?
Yes, several open-weight models now ship with context windows in the hundreds of thousands of tokens, though effective recall still varies by model rather than by license type. See our open-source vs closed-source LLM comparison for the fuller picture.
Where to go next
For more on model internals and efficiency tradeoffs, see our guides to parameter-efficient fine-tuning, quantized models, and open-source vs closed-source LLMs.