{"post":{"slug":"pipeline-parallelism-explained-2026","title":"Pipeline Parallelism: Splitting a Model Across Machines","excerpt":"Give each machine a contiguous block of layers and pass activations along the chain. It scales across ordinary networks, and it introduces idle time you have to engineer away.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span> model too large for a single machine has to span several, and the network between machines is far slower than the links between GPUs inside one. That constraint rules out strategies that communicate constantly, and it is what pipeline parallelism is built around.\n\nSplit the model by depth. Machine one holds the first block of layers, machine two the next, and so on. Data flows through the chain, and communication happens only where one stage hands off to the next.\n\n## What changed in 2026\n\n- **Hybrid parallelism became the default for very large models.** Tensor parallelism within nodes, pipeline across them, settled as the standard configuration.\n- **Scheduling improved.** Better micro-batch interleaving reduced bubble time compared with naive implementations.\n- **Stage balancing got tooling.** Automatic partitioning that accounts for uneven layer cost replaced hand-tuned splits.\n- **Disaggregated serving emerged as a related idea.** Separating prefill from decode across different hardware borrowed the same stage-boundary thinking.\n\n## Why it scales across a network\n\nThe defining property is communication volume. At a stage boundary you pass activations — the intermediate representation for the tokens in flight — and nothing else. That happens once per stage, not once per layer.\n\nCompare with tensor parallelism, which requires a collective operation after every parallelised layer. For a deep model that is a large multiple more communication events.\n\nSo pipeline parallelism tolerates ordinary datacentre networking, where tensor parallelism does not. That is why the two get combined rather than treated as alternatives: use the fast intra-node links for the chatty strategy, and the slower inter-node links for the sparing one.\n\n## The bubble\n\nThe cost is idle time, and it follows directly from the structure.\n\nStage one processes the first batch and hands it to stage two. While stage two works, stage one has nothing to do unless something else arrives. At the start of a batch, later stages wait for work to reach them; at the end, earlier stages wait for the pipeline to drain. That idle time is the **bubble**.\n\nWith one batch in flight, the bubble is severe — most devices are idle most of the time, which defeats the purpose of having them.\n\n**Micro-batching** is the fix. Split the batch into smaller pieces and feed them in sequence. While stage two works on piece one, stage one starts piece two. The pipeline fills, and every stage stays busy for most of the run.\n\nMore micro-batches means a smaller bubble as a proportion of total time, and more scheduling overhead and more memory for activations in flight. The tuning is finding where those cross.\n\n| Micro-batches | Bubble | Overhead |\n| --- | --- | --- |\n| One | Severe | Minimal |\n| Few | Noticeable | Low |\n| Many | Small | Rising memory and scheduling |\n\n## Balancing the stages\n\nThe pipeline runs at the speed of its slowest stage. Uneven partitioning means fast stages waiting on a slow one, and the idle time is pure waste.\n\nBalancing is harder than dividing layer count equally, because layers differ in cost. The embedding and output layers are frequently much more expensive than a middle transformer block, and putting them on the same stage as an equal number of ordinary layers unbalances the pipeline badly.\n\nProfile actual per-layer cost and partition on that, rather than on count. Automated partitioning tools do this; hand-tuned splits that assume uniform layers are a common source of unexplained underperformance.\n\n## Common mistakes\n\n- **One batch in flight.** Maximal bubble; most hardware idle.\n- **Equal layer counts per stage.** Layers are not equally expensive.\n- **Using it within a node.** Tensor parallelism does better where the links are fast.\n- **Expecting lower single-request latency.** A token still passes through every stage sequentially.\n- **Too many micro-batches.** Memory for in-flight activations grows.\n- **Not profiling per-stage time.** The slowest stage sets the pace and is not obvious.\n\n## FAQ\n\n**Does this reduce latency for one request?**\nNo. That request still traverses every stage in order, plus the network hops between them — so single-request latency is slightly worse. What improves is throughput, because stages work concurrently on different pieces.\n\n**How do I choose the number of stages?**\nThe fewest that make the model fit. Each additional stage adds a boundary, a network hop, and bubble. Stages are a cost to be minimised subject to the memory constraint.\n\n**Can I combine it with tensor parallelism?**\nYes, and for very large models you should. Tensor parallelism within each node, pipeline across nodes — see [tensor parallelism](/blog/tensor-parallelism-explained-2026).\n\n**What happens if one stage fails?**\nThe whole pipeline stalls, since every request needs every stage. Fault tolerance in a pipeline-parallel deployment is a real design concern rather than an afterthought.\n\n## Where to go next\n\nFor the within-node strategy that pairs with this, read [tensor parallelism](/blog/tensor-parallelism-explained-2026). For splitting the two phases of inference across hardware, [disaggregated serving](/blog/disaggregated-serving-2026), and for the scheduling layer above it, [continuous batching](/blog/continuous-batching-explained-2026).","hue":208,"metaTitle":"Pipeline Parallelism: Splitting a Model by Layers","metaDescription":"How pipeline parallelism assigns layer blocks to machines, what the pipeline bubble costs, and how micro-batching reduces idle time.","keywords":["pipeline parallelism","model sharding layers","pipeline bubble","micro batching","distributed model serving","multi node inference","stage partitioning","large model infrastructure 2026"],"tldr":["**Each stage holds complete consecutive layers** — so communication happens only at stage boundaries, not every layer.","**The bubble is the cost** — stages sit idle waiting for work, and micro-batching is how you shrink it.","**It scales across ordinary networking** — unlike tensor parallelism, which needs a fast interconnect.","**It improves throughput, not single-request latency** — a token still traverses every stage in sequence.","**Skip:** pipeline parallelism inside a single node with fast GPU links. Tensor parallelism does better there."],"readTime":"4 min read","wordCount":829},"related":[{"slug":"tensor-parallelism-explained-2026","title":"Tensor Parallelism: Splitting One Layer Across GPUs","excerpt":"When a model is too large for one GPU, you split it. Tensor parallelism divides individual layers across devices, which is fast and demands an interconnect most clusters do not have.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":203},{"slug":"adapter-merging-explained-2026","title":"Adapter Merging: Folding a Fine-Tune Back Into the Model","excerpt":"A trained adapter can stay separate and swappable, or be merged into the base weights for faster inference. You cannot have both, and the choice shapes your serving architecture.","category":"ai","date":"January 7, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":221},{"slug":"moe-routing-explained-2026","title":"MoE Routing Explained: How Experts Get Chosen","excerpt":"A mixture-of-experts model activates a fraction of its parameters per token. Here is how the router decides, why load balancing is hard, and what it means for serving.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":218}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"logical-replication-explained-2026","title":"Logical Replication: Replicating Changes, Not Blocks","excerpt":"Physical replication copies a byte-identical database. Logical replication streams row changes, which makes selective replication and cross-version upgrades possible — with sharper edges.","category":"code","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":320},"older":{"slug":"price-drop-refunds-2026","title":"Price Drop Refunds: Getting Money Back After You Bought","excerpt":"The item you bought last week is cheaper today. Many retailers will refund the difference if you ask within a window, and almost nobody asks.","category":"guides","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":161}}