The attention mechanism is the calculation a transformer uses to decide how much each token in a sequence should influence the representation of every other token. Rather than treating all context equally, attention produces a learned, per-token weighting: when processing the word "it" in a sentence, attention lets the model figure out, mathematically, which earlier noun "it" most likely refers to, and weight that connection accordingly.
What changed in 2026
- Efficient attention approximations became the default at scale. Flash Attention-style kernels and sparse or windowed attention variants are now standard in production training and inference stacks, cutting memory use without materially hurting quality.
- Multi-query and grouped-query attention displaced full multi-head attention in most new architectures, sharing key/value projections across heads to reduce memory bandwidth during inference.
- Interpretability research pushed back on naive readings of attention weights. The field increasingly treats raw attention maps as a partial signal rather than a full explanation of model behavior.
Queries, keys, and values
Every token's embedding gets projected into three separate vectors using learned weight matrices: a query, a key, and a value. The query represents what this token is "looking for." The key represents what each token (including itself) "offers" as a match. The value is the actual content that gets passed along if a match is found. Attention computes a similarity score between the query of the token being processed and the key of every other token, turns those scores into weights with a softmax function, and then produces the output as a weighted sum of every token's value vector, using those weights.
This is why the mechanism is called "scaled dot-product attention" — the similarity score is a dot product between query and key vectors, scaled down before the softmax to keep gradients stable during training.
Multi-head attention
Rather than running this process once, transformers run it several times in parallel, each with its own separate learned query, key, and value projections — these are the "heads." Each head can, in principle, specialize in a different kind of relationship: one might track subject-verb agreement, another might track long-range topical relevance. The outputs of all heads are concatenated and projected back down to the model's working dimension. This parallel structure is part of why the term self-attention (covered separately in what is self-attention) is often used interchangeably with attention in transformer contexts, even though self-attention specifically refers to attention within a single sequence rather than between two different sequences.
Attention variants compared
| Variant |
Key idea |
Tradeoff |
| Full multi-head attention |
Separate keys/values per head |
Highest quality, highest memory cost |
| Multi-query attention |
Single shared key/value across all heads |
Much less memory, small quality cost |
| Grouped-query attention |
Shared key/value across groups of heads |
Middle ground, now most common |
| Sliding-window / sparse attention |
Each token only attends to a local window |
Scales to long sequences, misses some long-range links |
Why attention is expensive
Because every token computes a relevance score against every other token, the compute and memory cost of full attention grows quadratically with sequence length: doubling the context length roughly quadruples the attention computation. This is the central bottleneck behind long-context model engineering, and it is why grouped-query attention, sliding windows, and specialized kernels exist — they exist specifically to make longer contexts computationally survivable.
FAQ
What is the difference between attention and self-attention?
Self-attention is attention applied within a single sequence, where queries, keys, and values all come from the same input. General "attention" can also apply across two different sequences, as in the original encoder-decoder translation setup.
Why does attention need three separate projections instead of one?
Because "what I am looking for" (query), "what I offer as a match" (key), and "what I actually contribute if matched" (value) are functionally different roles, and giving the model separate learned projections for each lets it represent more nuanced relationships than a single shared vector could.
Does more attention heads always mean a better model?
Not necessarily. More heads add representational flexibility but also add parameters and compute; architecture choices balance head count against layer depth and width for the given parameter budget.
Can attention weights tell you why a model produced a certain output?
Only partially. Attention weights show where the model looked, not the full computation that determined its output — later layers and the feedforward blocks also shape the final result.
Where to go next