An LLM request has two phases that could hardly be less alike. Processing the prompt is a large, parallel, compute-hungry operation. Generating the response is a long sequence of small steps, each limited by how fast weights can be read from memory.
Run both on the same machine and each interferes with the other. The usual symptom is that one user pasting a long document causes everyone else's tokens to stutter.
What changed in 2026
- Disaggregation moved from research to deployment. Large-scale serving operators began separating the phases in production rather than theorising about it.
- Chunked prefill became the mainstream alternative. Splitting long prompts into pieces that interleave with decode addressed much of the interference without new architecture.
- Heterogeneous hardware became viable. Different GPU types for the two phases started making economic sense at scale.
- Cache transfer stayed the constraint. Moving KV state between pools remained the technical requirement that gates the whole approach.
Two phases, opposite needs
|
Prefill |
Decode |
| Work shape |
All prompt tokens at once |
One token at a time |
| Bound by |
Compute |
Memory bandwidth |
| Parallelism |
High |
Low per sequence |
| Duration |
One burst |
Many small steps |
| Benefits from |
Raw FLOPs |
Fast memory, high batch |
| Latency metric |
Time to first token |
Time per output token |
Because decode is bandwidth-bound and does little arithmetic per step, batching many sequences together is what keeps a GPU busy during it — the weights are read once and applied to every sequence in the batch. Prefill needs no such help; a single long prompt already saturates the compute.
Running both on one pool means compromising. The batch composition that suits decode is disrupted every time a large prefill arrives, and the prefill itself is delayed by decode steps in progress.
The interference problem
The concrete symptom: a single long prompt degrades latency for every concurrent user.
While the GPU processes a large prefill, decode steps for other sequences wait. Those users see their token stream pause. It is not a fairness bug in the scheduler; the hardware is genuinely busy with someone else's prompt.
Two responses exist.
Chunked prefill splits a long prompt into pieces and interleaves them with decode steps. The prefill takes slightly longer overall and stops monopolising the device. This works within a single pool, requires no architectural change, and is the right answer for most deployments.
Disaggregation goes further: separate machines for each phase. Prefill machines do nothing but process prompts; decode machines do nothing but generate. Neither interferes with the other, and each can be sized and provisioned independently.
What disaggregation costs
The KV cache produced during prefill must reach the machine doing decode. That is a transfer of real size — proportional to prompt length — on the critical path of every request.
Which makes interconnect bandwidth the gating requirement. With a fast link between pools the transfer is a small addition to time to first token. Over ordinary networking it can exceed the benefit entirely, the same constraint that limits tensor parallelism across nodes.
The operational cost is also real: two pools to scale, two failure domains, a routing layer, and capacity planning that must keep the ratio between them correct as traffic shape changes. A workload that shifts toward longer prompts needs more prefill capacity, and a mismatched ratio leaves one pool idle while the other queues.
That is why this is a scale technique. Below a certain size, one pool with chunked prefill delivers most of the benefit for none of the complexity.
Common mistakes
- Disaggregating too early. Chunked prefill first; it is far simpler.
- Ignoring transfer cost. Over slow links it dominates.
- Fixed pool ratios. Traffic shape changes; the ratio must follow.
- Measuring only aggregate throughput. The point is latency isolation, which aggregate numbers hide.
- Not separating the two latency metrics. Time to first token and time per output token are affected differently.
- Assuming it helps a low-traffic deployment. With little concurrency there is no interference to remove.
FAQ
Should I do this?
Almost certainly not unless you operate serving at substantial scale. Enable chunked prefill, measure whether interference remains a problem, and only then consider separate pools.
Can I use different GPUs for each phase?
That is one of the main attractions — prefill benefits from compute-dense hardware, decode from memory bandwidth and capacity. Matching hardware to phase is a real cost saving at scale.
How does it interact with batching?
Favourably. Decode pools can maintain large, stable batches without prefills disrupting composition, which is exactly what continuous batching wants.
What about the two latency metrics?
They separate cleanly, which is a diagnostic benefit in itself — you can see which pool is constrained. See TTFT vs TPOT.
Where to go next
For the scheduling layer this restructures, read continuous batching. For the latency metrics it separates, TTFT vs TPOT, and for the parallelism strategies it composes with, pipeline parallelism.