Every sequence a language model generates carries a KV cache — stored attention keys and values for every token processed so far. The cache grows as the sequence grows, and the naive way to handle that is to reserve enough contiguous memory for the longest sequence you allow.
Do that and a request that generates fifty tokens holds memory reserved for four thousand. Most of the GPU's memory sits allocated to sequences that will never use it, and concurrency collapses long before compute is saturated.
What changed in 2026
- Paging became standard. Every serious serving framework adopted block-based KV cache allocation, and non-paged implementations became the exception.
- Prefix sharing built on it. Block sharing turned into a first-class feature for workloads with common prompt prefixes — see prefix caching.
- Longer contexts raised the stakes. As windows grew, the waste from over-reservation grew proportionally, which made the technique more valuable rather than less.
- Cache quantisation arrived alongside. Compressing cache entries stacked with paging to push concurrency further.
The waste, concretely
The problem is the gap between reserved and used.
| Approach |
Reserved per sequence |
Wasted |
| Contiguous, max length |
Maximum allowed |
Everything beyond actual use |
| Contiguous, guessed length |
An estimate |
Over-reserve wastes, under-reserve fails |
| Paged blocks |
Blocks actually needed |
Less than one block per sequence |
Contiguous allocation forces an impossible choice. Reserve for the maximum and waste enormously. Reserve less and a sequence that grows past the reservation has nowhere to go — you must either move it, which is expensive, or fail the request.
Fragmentation makes it worse. Sequences of different lengths finishing at different times leave gaps that are individually too small to hold a new sequence, so memory sits free and unusable.
How paging fixes it
The idea is borrowed directly from operating system virtual memory, which solved this same problem decades ago.
Divide GPU memory into fixed-size blocks. When a sequence needs cache space, allocate a block. When it fills, allocate another. Maintain a table mapping each sequence's logical positions to physical blocks, which need not be adjacent.
Three consequences follow:
Allocation matches use. A sequence holds blocks for the tokens it has actually produced, plus at most one partially-filled block. Waste falls from potentially enormous to a fraction of one block per sequence.
Fragmentation stops mattering. Any free block can serve any sequence, because contiguity is no longer required. Gaps are usable.
Blocks can be shared. Two sequences with an identical prefix can point at the same physical blocks rather than each holding a copy. That is the mechanism behind prefix caching, and it is a substantial saving on workloads where every request begins with the same long system prompt.
What it buys, and what it does not
The gain is concurrency. More usable memory means more sequences in flight simultaneously, which — combined with continuous batching — raises throughput considerably.
It does not make the arithmetic faster. A single request on an idle GPU is not accelerated. This is a memory efficiency technique whose benefit appears entirely under load, which is worth knowing when a single-user benchmark shows no difference.
Block size is the one tuning parameter of consequence. Smaller blocks reduce waste and increase table overhead; larger blocks do the reverse. Defaults in mature frameworks are reasonable, and this is rarely where your problem is.
Common mistakes
- Building it yourself. Solved infrastructure; use a framework that has it.
- Benchmarking with one request. The benefit is a concurrency benefit.
- Assuming memory is no longer a limit. It raises the ceiling substantially; long contexts still consume real memory.
- Ignoring preemption metrics. Under pressure the scheduler evicts and recomputes sequences, which is wasted work and a signal you are past your concurrency limit.
- Not exploiting prefix sharing. Workloads with a common prompt prefix leave a large saving unclaimed by default in some configurations.
- Tuning block size first. Rarely the bottleneck; look at total cache demand instead.
FAQ
Does this apply if I use a hosted API?
Your provider is running it, and you benefit without configuring anything. The practical lever on your side is prompt structure — a stable prefix enables sharing, per prefix caching.
How much concurrency does it add?
Depends entirely on how much your previous allocation over-reserved, which depends on the spread between your maximum allowed length and typical actual length. Workloads with high variance gain most.
Does it interact with quantisation?
Yes, and favourably. Quantising cache entries shrinks each block's footprint, so paging plus cache quantisation compounds — see KV cache quantization.
What happens when blocks run out?
The scheduler preempts a sequence, freeing its blocks, and recomputes it later. That is strictly wasted work, so rising preemption is the signal to reduce concurrency or add memory.
Where to go next
For the cache structure this manages, read KV cache explained. For the scheduler above it, continuous batching, and for the sharing it enables, prefix caching.