The context window is the single most misunderstood spec in an LLM datasheet. People treat it like storage — a bigger number must be strictly better. In practice it is a shared budget that your prompt, any retrieved data, the conversation history, and the model's own output all draw from at the same time. Understanding that shared budget is the difference between prompts that are cheap and reliable and prompts that are expensive and vague.
What changed in 2026
- Million-token windows became common. Several flagship models now advertise very large context lengths, but real-world quality often degrades well before the advertised maximum.
- Prompt caching matured. Providers now let you cache stable prefixes (system prompts, long documents) so repeated calls do not re-pay for the same tokens — a major cost lever for long-context apps.
- Benchmarks got honest about degradation. Newer long-context evaluations measure retrieval accuracy across the full window, exposing how much models drop toward the middle of very long inputs.
What the window actually holds
The context window is measured in tokens — chunks of text roughly three-quarters of a word in English. Everything the model sees or produces in a single call shares that budget:
- The system prompt and instructions
- The user's message
- Any retrieved documents or tool outputs you inject
- The full prior conversation, if you resend it
- The generated response itself
If a model has a 128k-token window and you fill 120k with input, only about 8k remain for the answer. Running out of output room mid-generation is a common, avoidable failure.
Tokens, cost, and latency
| Factor |
How the window affects it |
| Cost |
You pay per input and output token; larger prompts cost more on every single call |
| Latency |
More input tokens means more to process, so time-to-first-token rises |
| Reliability |
Very long contexts can bury the relevant part, hurting accuracy |
| Output room |
Input and output compete; a huge prompt leaves less space to answer |
Because input tokens dominate cost in retrieval-heavy apps, trimming context is often the highest-leverage optimization you can make. Caching stable prefixes and pairing the model with embeddings for targeted retrieval usually beats simply enlarging the prompt.
Bigger is not automatically better
Two effects push back against the "just use the whole window" instinct:
- Lost in the middle. Models reliably attend to the beginning and end of a long context, but accuracy for facts placed in the middle tends to sag. Put the most important material near the edges.
- Distraction. Irrelevant tokens are not free — they can pull the model off-topic and dilute attention. A tight, well-retrieved 4k prompt frequently outperforms a sprawling 100k one.
This is why retrieval-augmented generation exists: fetch only the passages that matter, rank them, and place them deliberately, rather than trusting the model to find the needle in a giant haystack.
Managing the budget in practice
- Measure before you optimize. Log token counts per call so you know where the budget actually goes.
- Summarize old turns. In long conversations, compress earlier history rather than resending it verbatim.
- Retrieve, do not dump. Use search to pull the relevant chunks; reserve raw long-context stuffing for cases where the whole document genuinely matters.
- Reserve output space. Always leave enough headroom for a complete response, plus any reasoning tokens the model needs.
FAQ
How many words fit in a context window?
As a rough rule, one token is about 0.75 English words, so a 128k-token window holds roughly 90,000–100,000 words of combined input and output. Code and non-English text tokenize less efficiently.
Does a bigger context window replace retrieval?
No. Large windows make it possible to include more, but retrieval still improves accuracy and cost by feeding the model only what is relevant. The two are complementary.
What happens if I exceed the limit?
The API rejects the request or truncates it, depending on the provider. Silent truncation is dangerous because the model may answer from partial input, so validate token counts before sending.
Why does accuracy drop on long inputs?
Attention is finite and unevenly distributed. Information in the middle of a very long context competes with everything around it, so models recall it less reliably than material at the start or end.
Where to go next