Most discussion of model compression concerns the weights. At long context lengths that is the wrong target, because the attention cache grows with every token in every concurrent sequence while the weights stay fixed.
Run enough concurrent sequences at enough context length and the cache becomes the largest consumer of GPU memory. Compressing it is then the lever that matters.
What changed in 2026
- Long contexts made the cache the dominant term. As typical prompt lengths grew, cache memory overtook weights as the binding constraint in many serving configurations.
- Cache quantisation reached mainstream frameworks. Configurable cache precision became a supported option rather than a research patch.
- Asymmetric schemes emerged. Treating keys and values differently, based on their differing sensitivity, improved the quality-versus-compression trade.
- It stacked with paging. Quantised entries in paged blocks compounded, pushing achievable concurrency substantially higher.
Why the cache dominates
Cache size scales with three things: context length, number of concurrent sequences, and model dimensions. The first two are exactly what you want to increase.
| Scenario |
Cache pressure |
| Short prompts, few users |
Negligible |
| Short prompts, many users |
Moderate |
| Long prompts, few users |
Moderate |
| Long prompts, many users |
Dominant |
That bottom row is where most production serving lands, and it is where the cache exceeds the weights by a wide margin. Compressing weights there saves a fixed amount once; compressing the cache saves proportionally to your concurrency, which is the number you are trying to raise.
The consequence is a hard ceiling on concurrency that has nothing to do with compute. The GPU may be far from saturated while the scheduler refuses new sequences because there is no cache space — see paged attention for how that space is managed.
What compression buys
Storing cache entries at lower precision reduces their footprint proportionally. Halving the precision roughly halves the memory per token, which roughly doubles how many sequences fit.
The benefit is entirely a concurrency benefit. A single request on an idle GPU is not faster — it may be marginally slower, since values are compressed on write and decompressed on read. Under load, where memory rather than compute is the constraint, the extra concurrency raises throughput substantially.
There is a secondary effect worth knowing: generation is memory-bandwidth-bound, so a smaller cache means less data moved per token, which can improve decode speed independently of the concurrency gain.
Where quality actually degrades
The useful finding is that keys and values behave differently under compression.
Keys determine which previous tokens receive attention. Errors there change the attention pattern itself, and the effect can compound across layers.
Values determine what is retrieved from the attended positions. Errors there produce a slightly noisier retrieved representation, which is more forgiving.
So keys tolerate compression worse than values, and schemes that keep keys at higher precision while compressing values more aggressively get a better trade than treating both identically.
The degradation is also task-dependent in a specific way: it concentrates in long-context recall. Short conversational exchanges barely notice. Tasks requiring precise retrieval of a detail from far back in a long context notice considerably. If your workload is the latter, evaluate on exactly that — a general benchmark will understate the cost, and needle-in-a-haystack tests are the relevant shape.
Common mistakes
- Applying it to short-context workloads. Little cache to compress; the complexity buys nothing.
- Evaluating on short tasks. Understates degradation that appears at long contexts.
- Compressing keys as aggressively as values. Keys are more sensitive.
- Expecting single-request speedup. The benefit is concurrency under load.
- Not measuring preemption. If sequences are still being evicted, you have not gained enough headroom.
- Confusing it with weight quantisation. Different memory, different tradeoffs, and they compose.
FAQ
How much quality do I lose?
Small for moderate compression on typical conversational workloads, and materially larger for precise long-context recall. Measure on your own task and context length rather than trusting a published figure.
Does it stack with weight quantisation?
Yes, and they address different memory. Weight quantisation lowers the fixed footprint; cache quantisation lowers the per-sequence footprint. Together they raise concurrency more than either alone — see quantization explained.
Is it available on hosted APIs?
Providers make these choices internally. It is a lever for self-hosted serving; on a hosted API your equivalent levers are prompt length and caching.
What compression level should I use?
Start moderate, measure on a long-context evaluation, and increase only while quality holds. The curve is not linear — there is usually a point where degradation accelerates noticeably.
Where to go next
For the memory system this compresses, read paged attention and KV cache explained. For evaluating the long-context cost, needle-in-a-haystack tests.