A transformer needs to know where each token sits in the sequence. Rotary position embedding encodes that by rotating the query and key vectors by an amount determined by position, so relative distances are represented directly in the attention computation.
It works well and it is trained on a particular range of positions. Feed the model a sequence longer than anything it saw during training and the rotations reach angles it has no experience of. Performance does not degrade gently — it tends to fall off sharply.
What changed in 2026
- Extension became standard practice. Nearly every long-context model reached its advertised length through some form of extension rather than training there from scratch.
- Frequency-aware methods displaced naive interpolation. Approaches treating different frequency components differently preserved short-sequence quality much better.
- Extension plus brief training became the norm. A small amount of continued training on long sequences recovers most of what scaling costs.
- Evaluation caught up. Testing at the extended length, rather than assuming it works, became expected — see needle-in-a-haystack tests.
The interpolation idea
The core trick is straightforward. If a model was trained to handle positions up to some length and you want twice that, divide every position index by two before computing the rotation. Position 8000 is treated as position 4000, which the model has seen.
The sequence now fits within the familiar range. Nothing is out of distribution.
The cost is resolution. Two tokens that were four positions apart are now treated as two apart, so fine-grained positional distinctions compress. The model becomes slightly less able to distinguish nearby positions precisely, which shows up as degradation on tasks requiring exact local ordering.
The other cost is that this applies uniformly, including to short sequences. A model with naive interpolation applied is worse on short inputs than it was before, which is a poor trade if most of your traffic is short.
Why frequency matters
Rotary embeddings use many frequency components. High-frequency components encode fine local distinctions; low-frequency ones encode long-range position.
Naive interpolation scales all of them equally. That is unnecessary — the high-frequency components were not the problem, since local relationships are the same at position 8000 as at position 400. It is the low-frequency components, encoding absolute long-range position, that go out of distribution.
Frequency-aware methods scale the low-frequency components while leaving high-frequency ones largely alone. Long-range positions get mapped into the familiar range, local resolution is preserved, and short-sequence performance suffers much less.
| Approach |
Long context |
Short context |
Training needed |
| No extension |
Fails past training length |
Unaffected |
None |
| Naive interpolation |
Works |
Degraded |
None, but better with |
| Frequency-aware scaling |
Works better |
Largely preserved |
None, but better with |
| Extension plus fine-tuning |
Best |
Preserved |
Some |
Extension is not free
An extended context window costs money and attention quality at every length you use it at.
Attention computation grows with sequence length. Memory for the KV cache grows linearly with context and directly caps concurrency. And model attention across a very long context remains uneven regardless of extension — the middle of a long input gets less weight than the ends.
Which means "we extended the context, so we can stop doing retrieval" is usually wrong. Retrieving relevant material into a moderate context frequently outperforms stuffing everything into a large one, on both cost and quality — see long context vs RAG.
Extension is worth doing when your inputs genuinely exceed what retrieval can chunk sensibly. It is not a substitute for deciding what belongs in the window.
Common mistakes
- Assuming extension works without testing. Evaluate at the extended length.
- Naive interpolation on a short-context workload. Degrades your common case.
- Extension without any continued training. Leaves recoverable quality on the table.
- Treating extended length as free capacity. Costs attention, memory, and concurrency.
- Abandoning retrieval after extending. Frequently a quality regression.
- Testing only single-fact recall. Multi-fact reasoning degrades first.
FAQ
How far can a model be extended?
Substantially, and quality degrades progressively rather than cutting off. The practical limit is where your evaluation shows unacceptable loss, which requires measuring rather than assuming.
Do I need to do this myself?
Almost certainly not — models ship with their advertised context already extended. This matters for understanding why long-context performance differs from short-context performance.
Does it affect short-sequence quality?
With naive methods, yes, noticeably. With frequency-aware methods, much less. This is the main reason the naive approach fell out of use.
How do I evaluate an extended model?
At the lengths you actually use, on multi-fact tasks rather than single-fact retrieval — see needle-in-a-haystack tests.
Where to go next
For the broader context-length question, read context extension. For evaluating long-context capability, needle-in-a-haystack tests, and for the memory cost, KV cache quantisation.