The context window grew faster than the ability to use it. A model advertising a million tokens will genuinely accept a million tokens without erroring. What it will not do is treat token nine hundred thousand with the same care as token five hundred. That gap between capacity and reliable use is what practitioners started calling context rot, and it is the reason large-window models keep disappointing teams who filled the window because they could.
What changed in 2026
- Simple retrieval benchmarks stopped being convincing. Needle-in-a-haystack tests, where a single fact is hidden in filler, are close to saturated. Models pass them and still fail on real long-document reasoning.
- Multi-fact evaluation replaced single-fact. The harder and more realistic test asks the model to combine several facts scattered through a long input, and scores there fall off much earlier.
- Distractor sensitivity got quantified. Adding plausible-but-irrelevant passages degrades accuracy more than adding an equal volume of obvious filler, which has direct implications for how retrieval systems select chunks.
- Context engineering became a named discipline. Deciding what not to include turned into a recognized skill rather than an afterthought.
Where quality actually falls off
| Prompt fill level |
Typical behavior |
Practical guidance |
| Under 25 percent of window |
Reliable across positions |
Safe working range |
| 25 to 50 percent |
Mild middle-position weakness |
Put critical material first or last |
| 50 to 75 percent |
Noticeable multi-fact degradation |
Split the task or retrieve harder |
| Over 75 percent |
Unreliable for reasoning; recall only |
Treat as an archive, not a workspace |
These bands vary by model and task, and you should measure your own rather than trusting the table. But the shape is consistent everywhere: degradation begins well before the window is full, and it hits reasoning across the context earlier than it hits finding something in the context.
Keeping long prompts sharp
The first move is counterintuitive: retrieve less. Teams frequently respond to a wrong answer by increasing the number of retrieved chunks, which adds distractors and often makes accuracy worse. Better retrieval beats more retrieval, which is the argument running through RAG chunking strategies.
Second, place deliberately. Put the instruction and the most critical document at the start or the end. Reference material that only sometimes matters goes in the middle, where a miss is cheapest.
Third, compact between turns. In a long agent run, periodically replace the accumulated transcript with a structured summary rather than letting it grow. This costs one model call and buys back the working range — the same logic behind the brief-passing patterns in AI agent handoff patterns.
Fourth, remember that reasoning tokens occupy the same window. A long thinking trace from a high test-time compute setting eats space your documents needed, so the two settings must be budgeted together.
Common mistakes
- Treating window size as a feature comparison. A model with a smaller window and better long-context behavior can outperform a larger-window model on the same task.
- Adding context to fix a wrong answer. Try removing irrelevant context first. It works more often than people expect.
- Never measuring your own decay curve. Published benchmarks use their documents, not yours. Run a small multi-fact test on your real corpus.
- Letting agent transcripts grow unbounded. Every tool result stays in context by default. Without compaction, a long run quietly slides into the unreliable band.
- Assuming caching fixes it. Prompt caching reduces the cost of a long prompt. It does nothing for the accuracy decay.
FAQ
Is context rot a bug that will be fixed?
It is being reduced, not eliminated. Architectural work keeps pushing the reliable range outward, but the pattern of degrading attention over very long inputs has proven persistent across generations.
Does it affect all models equally?
No. Long-context behavior varies substantially between models with identical advertised windows, which is why the advertised number is a poor comparison metric.
Is a large window still worth paying for?
Yes, for capacity — processing a large document at all, avoiding chunking complexity. Just do not assume the top of the range is a reasoning workspace.
How do I test my own decay curve?
Take a real long document, plant several facts that must be combined, and score the model at increasing fill levels. Twenty test cases is enough to see the shape.
Where to go next
Read context engineering explained for the discipline of deciding what goes in. For the retrieval side of keeping prompts small and relevant, see what is RAG and agentic RAG explained.