{"post":{"slug":"paged-attention-explained-2026","title":"Paged Attention: Fixing the KV Cache Memory Problem","excerpt":"Reserving contiguous memory for every sequence's maximum possible length wastes most of it. Borrowing virtual memory paging for the KV cache recovered that waste.","category":"ai","date":"January 6, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">E</span>very sequence a language model generates carries a KV cache — stored attention keys and values for every token processed so far. The cache grows as the sequence grows, and the naive way to handle that is to reserve enough contiguous memory for the longest sequence you allow.\n\nDo that and a request that generates fifty tokens holds memory reserved for four thousand. Most of the GPU's memory sits allocated to sequences that will never use it, and concurrency collapses long before compute is saturated.\n\n## What changed in 2026\n\n- **Paging became standard.** Every serious serving framework adopted block-based KV cache allocation, and non-paged implementations became the exception.\n- **Prefix sharing built on it.** Block sharing turned into a first-class feature for workloads with common prompt prefixes — see [prefix caching](/blog/prefix-caching-explained-2026).\n- **Longer contexts raised the stakes.** As windows grew, the waste from over-reservation grew proportionally, which made the technique more valuable rather than less.\n- **Cache quantisation arrived alongside.** Compressing cache entries stacked with paging to push concurrency further.\n\n## The waste, concretely\n\nThe problem is the gap between reserved and used.\n\n| Approach | Reserved per sequence | Wasted |\n| --- | --- | --- |\n| Contiguous, max length | Maximum allowed | Everything beyond actual use |\n| Contiguous, guessed length | An estimate | Over-reserve wastes, under-reserve fails |\n| Paged blocks | Blocks actually needed | Less than one block per sequence |\n\nContiguous allocation forces an impossible choice. Reserve for the maximum and waste enormously. Reserve less and a sequence that grows past the reservation has nowhere to go — you must either move it, which is expensive, or fail the request.\n\nFragmentation makes it worse. Sequences of different lengths finishing at different times leave gaps that are individually too small to hold a new sequence, so memory sits free and unusable.\n\n## How paging fixes it\n\nThe idea is borrowed directly from operating system virtual memory, which solved this same problem decades ago.\n\nDivide GPU memory into fixed-size blocks. When a sequence needs cache space, allocate a block. When it fills, allocate another. Maintain a table mapping each sequence's logical positions to physical blocks, which need not be adjacent.\n\nThree consequences follow:\n\n**Allocation matches use.** A sequence holds blocks for the tokens it has actually produced, plus at most one partially-filled block. Waste falls from potentially enormous to a fraction of one block per sequence.\n\n**Fragmentation stops mattering.** Any free block can serve any sequence, because contiguity is no longer required. Gaps are usable.\n\n**Blocks can be shared.** Two sequences with an identical prefix can point at the same physical blocks rather than each holding a copy. That is the mechanism behind prefix caching, and it is a substantial saving on workloads where every request begins with the same long system prompt.\n\n## What it buys, and what it does not\n\nThe gain is concurrency. More usable memory means more sequences in flight simultaneously, which — combined with [continuous batching](/blog/continuous-batching-explained-2026) — raises throughput considerably.\n\nIt does not make the arithmetic faster. A single request on an idle GPU is not accelerated. This is a memory efficiency technique whose benefit appears entirely under load, which is worth knowing when a single-user benchmark shows no difference.\n\nBlock size is the one tuning parameter of consequence. Smaller blocks reduce waste and increase table overhead; larger blocks do the reverse. Defaults in mature frameworks are reasonable, and this is rarely where your problem is.\n\n## Common mistakes\n\n- **Building it yourself.** Solved infrastructure; use a framework that has it.\n- **Benchmarking with one request.** The benefit is a concurrency benefit.\n- **Assuming memory is no longer a limit.** It raises the ceiling substantially; long contexts still consume real memory.\n- **Ignoring preemption metrics.** Under pressure the scheduler evicts and recomputes sequences, which is wasted work and a signal you are past your concurrency limit.\n- **Not exploiting prefix sharing.** Workloads with a common prompt prefix leave a large saving unclaimed by default in some configurations.\n- **Tuning block size first.** Rarely the bottleneck; look at total cache demand instead.\n\n## FAQ\n\n**Does this apply if I use a hosted API?**\nYour provider is running it, and you benefit without configuring anything. The practical lever on your side is prompt structure — a stable prefix enables sharing, per [prefix caching](/blog/prefix-caching-explained-2026).\n\n**How much concurrency does it add?**\nDepends entirely on how much your previous allocation over-reserved, which depends on the spread between your maximum allowed length and typical actual length. Workloads with high variance gain most.\n\n**Does it interact with quantisation?**\nYes, and favourably. Quantising cache entries shrinks each block's footprint, so paging plus cache quantisation compounds — see [KV cache quantization](/blog/kv-cache-quantization-2026).\n\n**What happens when blocks run out?**\nThe scheduler preempts a sequence, freeing its blocks, and recomputes it later. That is strictly wasted work, so rising preemption is the signal to reduce concurrency or add memory.\n\n## Where to go next\n\nFor the cache structure this manages, read [KV cache explained](/blog/kv-cache-explained-2026). For the scheduler above it, [continuous batching](/blog/continuous-batching-explained-2026), and for the sharing it enables, [prefix caching](/blog/prefix-caching-explained-2026).","hue":214,"metaTitle":"Paged Attention: Fixing KV Cache Memory Waste","metaDescription":"Why contiguous KV cache allocation wastes memory, how paging recovers it, and why block sharing makes prefix caching possible.","keywords":["paged attention","kv cache memory","vllm paging","memory fragmentation llm","block allocation inference","concurrency llm serving","prefix sharing","inference optimization 2026"],"tldr":["**Contiguous pre-allocation wastes most of the reserved memory** — sequences rarely reach the maximum length you reserved for.","**Paging allocates fixed blocks on demand** — memory grows with actual use rather than potential use.","**More usable memory means higher concurrency** — which is the throughput win, not faster arithmetic.","**Blocks can be shared across sequences** — identical prefixes point at the same physical blocks.","**Skip:** implementing this yourself. It is standard in serving frameworks and not worth rebuilding."],"readTime":"4 min read","wordCount":852},"related":[{"slug":"kv-cache-explained-2026","title":"KV Cache Explained: Why Long Chats Get Expensive","excerpt":"The key-value cache is what makes token generation fast and what makes long contexts consume so much memory. Here is how it works and how to control its size.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":205},{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":205},{"slug":"flash-attention-explained-2026","title":"Flash Attention Explained: Speed From Memory Movement","excerpt":"Flash attention makes transformers faster without changing the math. Here is what it actually optimizes, why memory bandwidth was the bottleneck, and when it matters.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":208}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"instruction-tuning-explained-2026","title":"Instruction Tuning: Turning a Text Predictor Into an Assistant","excerpt":"A base model continues text. It does not answer questions, follow instructions, or stop. Instruction tuning is the step that turns one into the other.","category":"ai","date":"January 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":210},"older":{"slug":"portfolio-turnover-cost-2026","title":"Portfolio Turnover: The Cost That Does Not Appear in the Fee","excerpt":"A fund's expense ratio is disclosed prominently. Its trading costs and the tax bill it generates for you are not, and for an active fund they can exceed the stated fee.","category":"finance","date":"January 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":290}}