A model too large for a single machine has to span several, and the network between machines is far slower than the links between GPUs inside one. That constraint rules out strategies that communicate constantly, and it is what pipeline parallelism is built around.
Split the model by depth. Machine one holds the first block of layers, machine two the next, and so on. Data flows through the chain, and communication happens only where one stage hands off to the next.
What changed in 2026
- Hybrid parallelism became the default for very large models. Tensor parallelism within nodes, pipeline across them, settled as the standard configuration.
- Scheduling improved. Better micro-batch interleaving reduced bubble time compared with naive implementations.
- Stage balancing got tooling. Automatic partitioning that accounts for uneven layer cost replaced hand-tuned splits.
- Disaggregated serving emerged as a related idea. Separating prefill from decode across different hardware borrowed the same stage-boundary thinking.
Why it scales across a network
The defining property is communication volume. At a stage boundary you pass activations — the intermediate representation for the tokens in flight — and nothing else. That happens once per stage, not once per layer.
Compare with tensor parallelism, which requires a collective operation after every parallelised layer. For a deep model that is a large multiple more communication events.
So pipeline parallelism tolerates ordinary datacentre networking, where tensor parallelism does not. That is why the two get combined rather than treated as alternatives: use the fast intra-node links for the chatty strategy, and the slower inter-node links for the sparing one.
The bubble
The cost is idle time, and it follows directly from the structure.
Stage one processes the first batch and hands it to stage two. While stage two works, stage one has nothing to do unless something else arrives. At the start of a batch, later stages wait for work to reach them; at the end, earlier stages wait for the pipeline to drain. That idle time is the bubble.
With one batch in flight, the bubble is severe — most devices are idle most of the time, which defeats the purpose of having them.
Micro-batching is the fix. Split the batch into smaller pieces and feed them in sequence. While stage two works on piece one, stage one starts piece two. The pipeline fills, and every stage stays busy for most of the run.
More micro-batches means a smaller bubble as a proportion of total time, and more scheduling overhead and more memory for activations in flight. The tuning is finding where those cross.
| Micro-batches |
Bubble |
Overhead |
| One |
Severe |
Minimal |
| Few |
Noticeable |
Low |
| Many |
Small |
Rising memory and scheduling |
Balancing the stages
The pipeline runs at the speed of its slowest stage. Uneven partitioning means fast stages waiting on a slow one, and the idle time is pure waste.
Balancing is harder than dividing layer count equally, because layers differ in cost. The embedding and output layers are frequently much more expensive than a middle transformer block, and putting them on the same stage as an equal number of ordinary layers unbalances the pipeline badly.
Profile actual per-layer cost and partition on that, rather than on count. Automated partitioning tools do this; hand-tuned splits that assume uniform layers are a common source of unexplained underperformance.
Common mistakes
- One batch in flight. Maximal bubble; most hardware idle.
- Equal layer counts per stage. Layers are not equally expensive.
- Using it within a node. Tensor parallelism does better where the links are fast.
- Expecting lower single-request latency. A token still passes through every stage sequentially.
- Too many micro-batches. Memory for in-flight activations grows.
- Not profiling per-stage time. The slowest stage sets the pace and is not obvious.
FAQ
Does this reduce latency for one request?
No. That request still traverses every stage in order, plus the network hops between them — so single-request latency is slightly worse. What improves is throughput, because stages work concurrently on different pieces.
How do I choose the number of stages?
The fewest that make the model fit. Each additional stage adds a boundary, a network hop, and bubble. Stages are a cost to be minimised subject to the memory constraint.
Can I combine it with tensor parallelism?
Yes, and for very large models you should. Tensor parallelism within each node, pipeline across nodes — see tensor parallelism.
What happens if one stage fails?
The whole pipeline stalls, since every request needs every stage. Fault tolerance in a pipeline-parallel deployment is a real design concern rather than an afterthought.
Where to go next
For the within-node strategy that pairs with this, read tensor parallelism. For splitting the two phases of inference across hardware, disaggregated serving, and for the scheduling layer above it, continuous batching.