The transformer is the architecture introduced in the 2017 paper "Attention Is All You Need," and it is still, nearly a decade later, the foundation underneath almost every mainstream language model. Its defining idea is processing an entire sequence of tokens at once and letting every token directly attend to every other token, rather than processing text one step at a time the way earlier recurrent networks did. That single change is what made large-scale parallel training practical and, ultimately, made today's large language models possible.
What changed in 2026
- Efficient attention variants became standard, not experimental. Grouped-query attention and sliding-window attention, once niche optimizations, are now default choices in most production model architectures to cut memory bandwidth without a meaningful quality hit.
- Decoder-only designs consolidated their dominance. Most frontier language models now use a decoder-only transformer rather than the original encoder-decoder design, reserving encoder-decoder architectures mainly for translation and some multimodal tasks.
- Positional encoding methods kept evolving. Rotary position embeddings (RoPE) and its extensions remain the dominant approach, with ongoing work on extending effective context length without retraining from scratch.
The core building blocks
A transformer layer has two main sub-components, applied repeatedly across many stacked layers: a self-attention block and a feedforward block. Self-attention lets each token gather information from other tokens in the sequence, weighted by relevance — see attention mechanism explained for the mechanics of how that weighting is computed. The feedforward block then transforms each token's representation independently, applying the same learned transformation to every position. Attention mixes information across positions; the feedforward layer processes each position on its own.
Wrapped around both blocks are two supporting mechanisms that make deep stacking possible: residual connections, which add each block's input back to its output so gradients can flow through many layers without vanishing, and layer normalization, which keeps activation values in a stable range throughout training.
Encoder-only, decoder-only, and encoder-decoder
| Variant |
How it works |
Common use |
| Encoder-only (e.g. BERT-style) |
Sees the full sequence at once, no masking |
Classification, embeddings, understanding tasks |
| Decoder-only (e.g. GPT-style) |
Each token only attends to earlier tokens |
Text generation, chat, most modern LLMs |
| Encoder-decoder (e.g. T5-style) |
Encoder processes input, decoder generates output attending to it |
Translation, summarization with distinct input/output |
Why transformers replaced recurrent networks
Recurrent neural networks (RNNs) processed text one token at a time, carrying a hidden state forward — which made them inherently sequential and hard to parallelize during training. Transformers process the whole sequence simultaneously, which lets training use the full parallelism of modern GPUs and made scaling to today's model sizes computationally feasible. The tradeoff is that self-attention's compute cost grows quadratically with sequence length, which is why efficient attention variants and context-length engineering remain an active area of work.
Positional encoding: the piece attention lacks
Self-attention on its own has no built-in notion of order — mathematically, it would treat a shuffled sentence identically to the original unless position information is injected separately. That is the job of positional encoding, covered in depth in positional encoding explained. Every practical transformer needs some mechanism for this; the original paper used fixed sinusoidal functions, and most current models use rotary embeddings instead.
FAQ
Do all language models use the same transformer design?
No. Most current LLMs use a decoder-only variant, but the underlying self-attention and feedforward building blocks are shared across nearly all variants.
Why do transformers need positional encoding if attention already looks at all tokens?
Because attention computes relevance between tokens without any inherent sense of their order — word order has to be added back in explicitly for the model to understand sequence structure.
What replaced RNNs, and why?
Transformers replaced RNNs mainly because self-attention parallelizes across an entire sequence during training, while RNNs process tokens sequentially and cannot be parallelized the same way.
Is a transformer the same thing as a large language model?
No. Transformer is the underlying architecture; a large language model is a specific transformer-based model trained at scale on a large dataset for language tasks.
Where to go next