Generating text one token at a time would be catastrophically slow if the model reprocessed the entire prompt for each new token. It does not. It stores the intermediate attention state — the keys and values — for every token it has already seen, and reuses them. That store is the KV cache, and it is the single reason autoregressive generation is practical.
It is also where your memory goes.
What changed in 2026
- Paged allocation became standard. Serving stacks adopted memory management that allocates cache in fixed blocks rather than contiguous per-request buffers, which sharply reduced fragmentation waste.
- Prompt caching went mainstream commercially. Providers exposed prefix reuse as a billing discount, which turned an internal optimization into something application developers design around.
- Cache compression matured. Techniques reducing the precision or the number of cached entries moved from research into production serving options.
- Long-context pricing reflected reality. As providers understood their own cache economics better, pricing for very long contexts began reflecting the memory occupancy rather than only token count.
Why it dominates memory
| Factor |
Effect on cache size |
| Context length |
Linear; twice the tokens is twice the cache |
| Batch size |
Linear; each concurrent request has its own |
| Model layers |
Linear; every layer caches separately |
| Attention head configuration |
Grouped-query designs cut it substantially |
| Numeric precision |
Halving precision roughly halves cache size |
| Output length |
Grows throughout generation |
The consequence that surprises people is that at long context the cache can exceed the model weights in memory. Weights are fixed — load them once and they are done. Cache scales with how much text is in flight across all concurrent users, so a server holding fifty long conversations may be spending most of its memory on cache rather than on the model.
That is why concurrency drops as context grows. A server that comfortably handles many short conversations handles far fewer long ones, using the same hardware, running the same model.
What you can actually do about it
If you are calling an API, the lever is prompt caching. When many requests share a long prefix — a system prompt, a document, a set of examples — providers can reuse the cached state rather than recomputing it, and they price that reuse at a discount. Structuring prompts so the shared part comes first and the variable part comes last is what makes this work. Put the changing content at the end.
Keep the context you actually need rather than everything available. Every token in the prompt occupies cache for the duration of the request, and that cost is real even when the model handles the length fine. This compounds with the accuracy decay covered in context engineering explained.
If you are self-hosting, the levers are the serving stack's cache configuration: paged allocation, cache quantization, and eviction policy for idle sessions. The memory arithmetic for capacity planning is in GPU memory estimate for LLMs.
Common mistakes
- Treating context length as free. It costs memory for the entire request, which limits how many requests fit.
- Putting variable content before the shared prefix. Defeats prompt caching entirely.
- Ignoring cache in capacity planning. Sizing on model weights alone underestimates memory badly.
- Very large batch sizes at long context. Multiplies cache demand and can trigger out-of-memory failures under load.
- Assuming cache persists indefinitely. Provider prompt caches expire; check the window before designing around them.
FAQ
Is the KV cache the same as prompt caching?
Prompt caching is a product feature built on the KV cache mechanism. The cache is the underlying data structure; prompt caching is reusing part of it across requests.
Does the cache affect output quality?
Not in itself. Cache quantization, which reduces precision to save memory, can affect quality slightly at aggressive settings.
Why does my throughput drop with long prompts?
Cache memory limits how many requests run concurrently. Longer prompts mean fewer concurrent requests, which reduces aggregate throughput even though per-request speed is similar.
Can I clear the cache mid-conversation?
On self-hosted stacks, yes, through session management. Via an API, the cache is managed by the provider and expires on its own schedule.
Where to go next
For memory planning, read GPU memory estimate for LLMs. For the attention mechanism that produces the cache, flash attention explained, and for keeping prompts small, context compression.