Context windows expanded from thousands of tokens to millions in a few years. The natural reading is that context is no longer a constraint and you can stop engineering around it.
That reading is wrong in three specific ways, and each of them costs money or quality.
What changed in 2026
- Very large windows became widely available. Million-token contexts moved from frontier capability to standard offering.
- The practical limits got documented. Attention unevenness, cost scaling, and concurrency effects became well-understood rather than surprising.
- Retrieval did not go away. Predictions that long context would replace retrieval did not hold, for cost and precision reasons.
- Caching became the enabling technology. Long stable prefixes are affordable only with prefix reuse — see prefix caching.
Three things that move against you
Cost. You pay for tokens sent, not for window size. Filling a million-token window costs a million tokens' worth of input on every request. A larger window is permission to spend more, not a discount.
Attention quality. Models attend unevenly across long inputs, with material near the beginning and end receiving more weight than the middle. Extension methods mitigate this and do not eliminate it — see RoPE scaling. Something buried in the middle of a very long context may as well not be there.
Concurrency. KV cache memory per sequence scales with context length, and cache memory is usually what caps how many sequences serve concurrently. Doubling typical context roughly halves achievable concurrency on the same hardware — a throughput cost that appears as a capacity problem rather than a context problem, per AI capacity planning.
| Window size |
Cost per full request |
Concurrency |
Attention reliability |
| Small |
Low |
High |
Good throughout |
| Moderate |
Moderate |
Moderate |
Good |
| Very large |
High |
Low |
Uneven; middle disadvantaged |
More context is not more signal
The quality argument people miss: adding irrelevant material to a context reduces precision.
A prompt containing the answer plus ten pages of related-but-wrong content is harder for the model than one containing the answer plus one page. The distractors compete for attention, and the model can and does attend to the wrong thing.
This is why retrieval into a moderate window frequently outperforms stuffing everything into a large one. Retrieval is a filtering step, and filtering improves the signal-to-noise ratio of what the model reads. Removing that step because the window is large trades a precision improvement for a convenience — see long context vs RAG.
The corollary is that retrieval quality matters more, not less, when you have room for many chunks. Retrieving fifty mediocre chunks because they fit is worse than retrieving five good ones.
Where large windows genuinely help
They are not useless — the benefit is real and narrower than advertised.
Documents that cannot be chunked sensibly. A legal contract where clauses reference each other across the whole document, or code where understanding requires several files at once, resists chunking. Here a large window solves something retrieval cannot.
Long conversations and agent runs. Accumulated history is inherently sequential and cannot be retrieved from selectively without losing the thread — though compaction bounds this better than raw window size.
Reducing pipeline complexity. For a low-volume application, putting everything in context and skipping a retrieval pipeline is a legitimate engineering simplification. That trade stops working as volume rises.
Few-shot examples. More examples in context genuinely helps some tasks, and this is a good use of additional room.
Common mistakes
- Filling the window because it exists. Cost with no quality benefit.
- Abandoning retrieval after extending. Loses the filtering benefit.
- Putting important content in the middle. Least attended position.
- Ignoring the concurrency cost. Shows up as a capacity problem later.
- Not caching a long stable prefix. The dominant avoidable cost.
- Evaluating at short lengths only. Degradation appears at depth.
- Assuming stated window equals usable window. Measure on your own tasks.
FAQ
Should I use the largest available window?
Use the smallest that holds what the task genuinely needs. Larger costs more per request and reduces concurrency for everyone.
Where should I put the most important content?
Beginning or end. Avoid burying critical material in the middle of a long context.
Does a big window remove the need for chunking?
No — you still need to decide what goes in, and that decision is what chunking and retrieval are for. It removes the need to fit within a tight budget, which is a different constraint.
How do I know my model's real usable context?
Test on your own tasks at the lengths you use, with multi-fact questions rather than single-fact lookups — see needle-in-a-haystack tests.
Where to go next
For the mechanism behind extension, read RoPE scaling. For the retrieval comparison, long context vs RAG, and for managing accumulated history, context compaction.