{"post":{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">M</span>ost discussion of model compression concerns the weights. At long context lengths that is the wrong target, because the attention cache grows with every token in every concurrent sequence while the weights stay fixed.\n\nRun enough concurrent sequences at enough context length and the cache becomes the largest consumer of GPU memory. Compressing it is then the lever that matters.\n\n## What changed in 2026\n\n- **Long contexts made the cache the dominant term.** As typical prompt lengths grew, cache memory overtook weights as the binding constraint in many serving configurations.\n- **Cache quantisation reached mainstream frameworks.** Configurable cache precision became a supported option rather than a research patch.\n- **Asymmetric schemes emerged.** Treating keys and values differently, based on their differing sensitivity, improved the quality-versus-compression trade.\n- **It stacked with paging.** Quantised entries in paged blocks compounded, pushing achievable concurrency substantially higher.\n\n## Why the cache dominates\n\nCache size scales with three things: context length, number of concurrent sequences, and model dimensions. The first two are exactly what you want to increase.\n\n| Scenario | Cache pressure |\n| --- | --- |\n| Short prompts, few users | Negligible |\n| Short prompts, many users | Moderate |\n| Long prompts, few users | Moderate |\n| Long prompts, many users | Dominant |\n\nThat bottom row is where most production serving lands, and it is where the cache exceeds the weights by a wide margin. Compressing weights there saves a fixed amount once; compressing the cache saves proportionally to your concurrency, which is the number you are trying to raise.\n\nThe consequence is a hard ceiling on concurrency that has nothing to do with compute. The GPU may be far from saturated while the scheduler refuses new sequences because there is no cache space — see [paged attention](/blog/paged-attention-explained-2026) for how that space is managed.\n\n## What compression buys\n\nStoring cache entries at lower precision reduces their footprint proportionally. Halving the precision roughly halves the memory per token, which roughly doubles how many sequences fit.\n\nThe benefit is entirely a concurrency benefit. A single request on an idle GPU is not faster — it may be marginally slower, since values are compressed on write and decompressed on read. Under load, where memory rather than compute is the constraint, the extra concurrency raises throughput substantially.\n\nThere is a secondary effect worth knowing: generation is memory-bandwidth-bound, so a smaller cache means less data moved per token, which can improve decode speed independently of the concurrency gain.\n\n## Where quality actually degrades\n\nThe useful finding is that keys and values behave differently under compression.\n\nKeys determine *which* previous tokens receive attention. Errors there change the attention pattern itself, and the effect can compound across layers.\n\nValues determine *what* is retrieved from the attended positions. Errors there produce a slightly noisier retrieved representation, which is more forgiving.\n\nSo keys tolerate compression worse than values, and schemes that keep keys at higher precision while compressing values more aggressively get a better trade than treating both identically.\n\nThe degradation is also task-dependent in a specific way: it concentrates in **long-context recall**. Short conversational exchanges barely notice. Tasks requiring precise retrieval of a detail from far back in a long context notice considerably. If your workload is the latter, evaluate on exactly that — a general benchmark will understate the cost, and [needle-in-a-haystack tests](/blog/needle-in-a-haystack-tests-2026) are the relevant shape.\n\n## Common mistakes\n\n- **Applying it to short-context workloads.** Little cache to compress; the complexity buys nothing.\n- **Evaluating on short tasks.** Understates degradation that appears at long contexts.\n- **Compressing keys as aggressively as values.** Keys are more sensitive.\n- **Expecting single-request speedup.** The benefit is concurrency under load.\n- **Not measuring preemption.** If sequences are still being evicted, you have not gained enough headroom.\n- **Confusing it with weight quantisation.** Different memory, different tradeoffs, and they compose.\n\n## FAQ\n\n**How much quality do I lose?**\nSmall for moderate compression on typical conversational workloads, and materially larger for precise long-context recall. Measure on your own task and context length rather than trusting a published figure.\n\n**Does it stack with weight quantisation?**\nYes, and they address different memory. Weight quantisation lowers the fixed footprint; cache quantisation lowers the per-sequence footprint. Together they raise concurrency more than either alone — see [quantization explained](/blog/quantization-explained-2026).\n\n**Is it available on hosted APIs?**\nProviders make these choices internally. It is a lever for self-hosted serving; on a hosted API your equivalent levers are prompt length and caching.\n\n**What compression level should I use?**\nStart moderate, measure on a long-context evaluation, and increase only while quality holds. The curve is not linear — there is usually a point where degradation accelerates noticeably.\n\n## Where to go next\n\nFor the memory system this compresses, read [paged attention](/blog/paged-attention-explained-2026) and [KV cache explained](/blog/kv-cache-explained-2026). For evaluating the long-context cost, [needle-in-a-haystack tests](/blog/needle-in-a-haystack-tests-2026).","hue":205,"metaTitle":"KV Cache Quantisation: More Sequences, Same Memory","metaDescription":"Why the KV cache outgrows model weights at long contexts, how quantising it raises concurrency, and where the quality cost actually appears.","keywords":["kv cache quantization","cache compression llm","long context memory","inference concurrency","key value cache precision","throughput optimization","memory bound inference","llm serving 2026"],"tldr":["**At long contexts the cache exceeds the weights** — which makes it the thing worth compressing.","**Halving cache precision roughly doubles concurrency** — the benefit is throughput, not speed per request.","**Keys tolerate compression worse than values** — asymmetric schemes exploit that.","**Quality loss concentrates in long-context recall** — short interactions barely notice.","**Skip:** cache quantisation on short-context workloads. There is little cache to compress."],"readTime":"4 min read","wordCount":805},"related":[{"slug":"paged-attention-explained-2026","title":"Paged Attention: Fixing the KV Cache Memory Problem","excerpt":"Reserving contiguous memory for every sequence's maximum possible length wastes most of it. Borrowing virtual memory paging for the KV cache recovered that waste.","category":"ai","date":"January 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":214},{"slug":"kv-cache-explained-2026","title":"KV Cache Explained: Why Long Chats Get Expensive","excerpt":"The key-value cache is what makes token generation fast and what makes long contexts consume so much memory. Here is how it works and how to control its size.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":205},{"slug":"continuous-batching-explained-2026","title":"Continuous Batching Explained: Why LLM Servers Got Fast","excerpt":"Static batching makes every request in a batch wait for the slowest one. Continuous batching lets finished sequences leave and new ones join mid-flight. The throughput difference is large.","category":"code","date":"August 11, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":305}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"failover-testing-2026","title":"Failover Testing: Finding Out Before Production Does","excerpt":"Automatic failover is configured and has never fired. Whether it works, how long it takes, and whether your application reconnects are all unknowns until you trigger it deliberately.","category":"code","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":308},"older":{"slug":"momentum-investing-explained-2026","title":"Momentum Investing: Buying What Has Already Gone Up","excerpt":"The idea that recent winners keep winning contradicts most investing intuition and has substantial historical support. It also crashes hard and unpredictably.","category":"finance","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":286}}