Positional encoding is the mechanism that tells a transformer where each token sits in a sequence. Self-attention, on its own, computes relevance between tokens using only their content — mathematically, it has no inherent notion of order, so shuffling the tokens in a sentence would produce the exact same attention computation unless position is injected some other way. Positional encoding solves that by adding position information into the token representations before, or during, the attention computation.
What changed in 2026
- RoPE variants remain the dominant choice, but 2025-2026 work refined how RoPE frequencies are scaled to extend context length after pretraining, without requiring a full retrain from scratch.
- ALiBi and other bias-based approaches kept a smaller but active niche, valued for extrapolating more gracefully to sequence lengths beyond what the model was trained on, at some cost to short-context quality.
- Hybrid and learned position schemes saw renewed research interest as labs pushed effective context windows into the millions of tokens, exposing limits in the original fixed-frequency approaches.
Why position needs to be added explicitly
Consider the sentences "the dog bit the man" and "the man bit the dog." They contain identical tokens; only the order differs, and the order completely changes the meaning. Self-attention's core computation — comparing queries and keys via dot products — treats input as an unordered set unless position is encoded into the vectors themselves. Every practical transformer architecture (see what is transformer architecture) needs some solution to this, and several different ones have become standard depending on the model family.
Positional encoding methods compared
| Method |
How it works |
Strength |
Weakness |
| Sinusoidal (original) |
Fixed sine/cosine functions added to embeddings |
Simple, no extra parameters |
Extrapolates poorly beyond training length |
| Learned absolute |
A trainable vector per position, added to embeddings |
Flexible, learns from data |
Fixed max length, cannot extrapolate |
| RoPE (rotary) |
Rotates query/key vectors by an angle based on position |
Encodes relative position naturally, widely adopted |
Extrapolation needs scaling tricks beyond training length |
| ALiBi |
Adds a distance-based penalty directly to attention scores |
Strong length extrapolation |
Slightly different quality tradeoffs at short range |
How RoPE works, conceptually
Rotary position embeddings take a different approach than simply adding a position vector to the token embedding. Instead, RoPE rotates the query and key vectors by an angle that depends on their position in the sequence, before the attention dot product is computed. The key property this produces: the dot product between a rotated query at position m and a rotated key at position n ends up depending only on their relative distance, m minus n, not their absolute positions. That relative framing is a large part of why RoPE generalizes better to sequences longer than the exact lengths seen during training compared to fixed absolute encodings, though it still requires careful frequency scaling to extend cleanly to much longer contexts.
Why this matters for context length
Every context-length claim you see attached to a model — 32K tokens, 128K tokens, a million tokens — is downstream of positional encoding choices as much as it is downstream of attention efficiency. A model trained on sequences up to a certain length has typically only seen positional signals within that range; pushing well beyond it without adaptation causes quality to degrade, sometimes sharply. Techniques for extending effective context after pretraining almost always involve rescaling or reinterpolating the positional encoding scheme rather than retraining the whole model.
FAQ
Why cannot a transformer just infer word order from context?
Self-attention's core computation is permutation-invariant by design — it treats the input as a set of tokens compared pairwise, with no inherent sequence structure, unless position is explicitly added.
Is RoPE used in most current large language models?
Yes, it is the most widely adopted approach in current mainstream transformer architectures, though some models use ALiBi or other variants, particularly when very long context extrapolation is the priority.
Does positional encoding count toward a model's parameters?
Sinusoidal and RoPE encodings add no learned parameters — they are fixed mathematical functions. Learned absolute positional embeddings do add parameters, one vector per possible position.
Can you extend a model's context length after it has already been trained?
Often yes, to a degree, using techniques that rescale the positional encoding's frequency or interpolate it for longer sequences — but quality on the extended range is rarely as strong as native training at that length.
Where to go next