{"post":{"slug":"disaggregated-serving-2026","title":"Disaggregated Serving: Separating Prefill From Decode","excerpt":"Processing a prompt and generating tokens have opposite hardware profiles. Running them on the same machine means one of them is always using the wrong resource.","category":"ai","date":"January 7, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span>n LLM request has two phases that could hardly be less alike. Processing the prompt is a large, parallel, compute-hungry operation. Generating the response is a long sequence of small steps, each limited by how fast weights can be read from memory.\n\nRun both on the same machine and each interferes with the other. The usual symptom is that one user pasting a long document causes everyone else's tokens to stutter.\n\n## What changed in 2026\n\n- **Disaggregation moved from research to deployment.** Large-scale serving operators began separating the phases in production rather than theorising about it.\n- **Chunked prefill became the mainstream alternative.** Splitting long prompts into pieces that interleave with decode addressed much of the interference without new architecture.\n- **Heterogeneous hardware became viable.** Different GPU types for the two phases started making economic sense at scale.\n- **Cache transfer stayed the constraint.** Moving KV state between pools remained the technical requirement that gates the whole approach.\n\n## Two phases, opposite needs\n\n| | Prefill | Decode |\n| --- | --- | --- |\n| Work shape | All prompt tokens at once | One token at a time |\n| Bound by | Compute | Memory bandwidth |\n| Parallelism | High | Low per sequence |\n| Duration | One burst | Many small steps |\n| Benefits from | Raw FLOPs | Fast memory, high batch |\n| Latency metric | Time to first token | Time per output token |\n\nBecause decode is bandwidth-bound and does little arithmetic per step, batching many sequences together is what keeps a GPU busy during it — the weights are read once and applied to every sequence in the batch. Prefill needs no such help; a single long prompt already saturates the compute.\n\nRunning both on one pool means compromising. The batch composition that suits decode is disrupted every time a large prefill arrives, and the prefill itself is delayed by decode steps in progress.\n\n## The interference problem\n\nThe concrete symptom: **a single long prompt degrades latency for every concurrent user.**\n\nWhile the GPU processes a large prefill, decode steps for other sequences wait. Those users see their token stream pause. It is not a fairness bug in the scheduler; the hardware is genuinely busy with someone else's prompt.\n\nTwo responses exist.\n\n**Chunked prefill** splits a long prompt into pieces and interleaves them with decode steps. The prefill takes slightly longer overall and stops monopolising the device. This works within a single pool, requires no architectural change, and is the right answer for most deployments.\n\n**Disaggregation** goes further: separate machines for each phase. Prefill machines do nothing but process prompts; decode machines do nothing but generate. Neither interferes with the other, and each can be sized and provisioned independently.\n\n## What disaggregation costs\n\nThe KV cache produced during prefill must reach the machine doing decode. That is a transfer of real size — proportional to prompt length — on the critical path of every request.\n\nWhich makes interconnect bandwidth the gating requirement. With a fast link between pools the transfer is a small addition to time to first token. Over ordinary networking it can exceed the benefit entirely, the same constraint that limits [tensor parallelism](/blog/tensor-parallelism-explained-2026) across nodes.\n\nThe operational cost is also real: two pools to scale, two failure domains, a routing layer, and capacity planning that must keep the ratio between them correct as traffic shape changes. A workload that shifts toward longer prompts needs more prefill capacity, and a mismatched ratio leaves one pool idle while the other queues.\n\nThat is why this is a scale technique. Below a certain size, one pool with chunked prefill delivers most of the benefit for none of the complexity.\n\n## Common mistakes\n\n- **Disaggregating too early.** Chunked prefill first; it is far simpler.\n- **Ignoring transfer cost.** Over slow links it dominates.\n- **Fixed pool ratios.** Traffic shape changes; the ratio must follow.\n- **Measuring only aggregate throughput.** The point is latency isolation, which aggregate numbers hide.\n- **Not separating the two latency metrics.** Time to first token and time per output token are affected differently.\n- **Assuming it helps a low-traffic deployment.** With little concurrency there is no interference to remove.\n\n## FAQ\n\n**Should I do this?**\nAlmost certainly not unless you operate serving at substantial scale. Enable chunked prefill, measure whether interference remains a problem, and only then consider separate pools.\n\n**Can I use different GPUs for each phase?**\nThat is one of the main attractions — prefill benefits from compute-dense hardware, decode from memory bandwidth and capacity. Matching hardware to phase is a real cost saving at scale.\n\n**How does it interact with batching?**\nFavourably. Decode pools can maintain large, stable batches without prefills disrupting composition, which is exactly what [continuous batching](/blog/continuous-batching-explained-2026) wants.\n\n**What about the two latency metrics?**\nThey separate cleanly, which is a diagnostic benefit in itself — you can see which pool is constrained. See [TTFT vs TPOT](/blog/ttft-vs-tpot-2026).\n\n## Where to go next\n\nFor the scheduling layer this restructures, read [continuous batching](/blog/continuous-batching-explained-2026). For the latency metrics it separates, [TTFT vs TPOT](/blog/ttft-vs-tpot-2026), and for the parallelism strategies it composes with, [pipeline parallelism](/blog/pipeline-parallelism-explained-2026).","hue":219,"metaTitle":"Disaggregated Serving: Prefill and Decode Apart","metaDescription":"Why prefill and decode have opposite hardware profiles, how separating them removes latency interference, and what transferring the KV cache costs.","keywords":["disaggregated serving","prefill decode separation","inference architecture","latency interference llm","kv cache transfer","heterogeneous gpu serving","inference scaling","llm infrastructure 2026"],"tldr":["**Prefill is compute-bound; decode is memory-bandwidth-bound** — opposite profiles competing for one machine.","**Mixing them causes latency interference** — a long prompt stalls everyone else's token generation.","**Separating them lets you size each pool independently** — and use different hardware for each.","**The cost is moving the KV cache between them** — which needs fast interconnect to be worth it.","**Skip:** disaggregation below substantial scale. The complexity only pays when both pools are large."],"readTime":"4 min read","wordCount":861},"related":[{"slug":"ai-capacity-planning-2026","title":"AI Capacity Planning: Sizing for Tokens, Not Requests","excerpt":"Requests per second is the wrong unit. Two workloads at identical request rates can differ tenfold in what they demand, because the token shape is what actually consumes capacity.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200},{"slug":"prefix-caching-explained-2026","title":"Prefix Caching: Not Recomputing the Same System Prompt","excerpt":"Every request starts with the same two thousand tokens of instructions, and by default the model processes them from scratch every single time. Caching that work is nearly free.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":197},{"slug":"batch-vs-streaming-inference-2026","title":"Batch vs streaming inference in 2026: which serving mode to use","excerpt":"Batch and streaming inference trade throughput against latency when you run a model. Here is what each mode is in 2026, why continuous batching blurred the line, and how to route your workload.","category":"ai","date":"June 27, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":290}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"bloat-monitoring-2026","title":"Bloat Monitoring: Measuring Space You Cannot See","excerpt":"A table twice the size of its live data reads twice as many pages for every scan. Bloat is invisible in ordinary metrics and degrades performance continuously until someone measures it.","category":"code","date":"January 7, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":320},"older":{"slug":"exactly-once-delivery-2026","title":"Exactly-Once Delivery: The Guarantee You Cannot Have","excerpt":"Message systems advertise exactly-once semantics. What is actually achievable is at-least-once delivery plus idempotent processing, which produces the same outcome by a different route.","category":"code","date":"January 7, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":312}}