{"post":{"slug":"tensor-parallelism-explained-2026","title":"Tensor Parallelism: Splitting One Layer Across GPUs","excerpt":"When a model is too large for one GPU, you split it. Tensor parallelism divides individual layers across devices, which is fast and demands an interconnect most clusters do not have.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span> model that does not fit in one GPU's memory has to be split across several. There are a few ways to cut it, and they have very different communication characteristics — which turns out to matter more than anything else about them.\n\nTensor parallelism makes the cut *within* each layer. Every GPU holds a slice of every layer's weights, every GPU participates in computing every token, and the partial results are combined after each layer.\n\n## What changed in 2026\n\n- **High-bandwidth interconnects spread.** More clusters shipped with fast intra-node GPU links, widening where tensor parallelism is practical.\n- **Hybrid strategies became standard.** Combining tensor parallelism within a node and pipeline parallelism across nodes settled as the default shape for very large models.\n- **Serving frameworks abstracted it.** Configuring parallelism moved from bespoke code to a configuration parameter, which made getting it wrong easier and cheaper to fix.\n- **Communication became the documented bottleneck.** Profiling showed collective operations dominating time in poorly-provisioned setups, which shifted attention to interconnect from raw compute.\n\n## How it differs from the alternatives\n\n| | Tensor parallel | Pipeline parallel | Data parallel |\n| --- | --- | --- | --- |\n| Split by | Within each layer | Across layers | Across batch |\n| Each GPU holds | A slice of everything | Some complete layers | A full copy |\n| Communication | Every layer | Between stages | Gradient sync |\n| Latency for one request | Reduced | Not reduced | Not reduced |\n| Interconnect demand | Very high | Moderate | Moderate |\n| Fits a model too big for one GPU | Yes | Yes | No |\n\nThe latency row is the reason to prefer tensor parallelism where you can afford it. Splitting the work of a single layer across four GPUs means four devices computing simultaneously on the same token, which genuinely reduces time to produce that token. Pipeline parallelism improves throughput by keeping stages busy across different requests; it does not make any individual request faster.\n\n## Communication is the constraint\n\nThe cost is unavoidable and structural. After each parallelised operation, partial results from every GPU must be combined before the next layer can proceed. That is a collective communication step, and it happens on every layer, for every token.\n\nWhich produces the defining property: **tensor parallelism is bounded by interconnect bandwidth, not by compute.**\n\nInside a node with a fast dedicated GPU interconnect, that communication is cheap enough to be worth it. Across nodes connected by ordinary networking, it is not — the collective operations dominate, and adding GPUs makes generation slower rather than faster. This is a genuinely counterintuitive outcome that surprises people scaling up.\n\nHence the standard hybrid: tensor parallelism *within* a node where the links are fast, pipeline parallelism *across* nodes where they are not. The parallelism strategy follows the hardware topology rather than the model architecture.\n\nDegree matters too. Higher tensor parallelism means smaller per-GPU slices and more communication relative to compute. Past a point the communication overhead exceeds the compute saving, and the optimal degree is usually smaller than the number of GPUs available.\n\n## Interaction with the rest of serving\n\nTensor parallelism composes with the batching and memory techniques that govern throughput. The KV cache is also split across devices, which is part of how it relieves memory pressure — see [paged attention](/blog/paged-attention-explained-2026) for the cache side and [continuous batching](/blog/continuous-batching-explained-2026) for the scheduling layer.\n\nOne practical consequence: a tensor-parallel deployment has a fixed GPU group. You cannot scale it by adding one GPU; you scale by adding another complete group. That granularity affects capacity planning and cost more than people expect — see [AI capacity planning](/blog/ai-capacity-planning-2026).\n\n## Common mistakes\n\n- **Using it across a slow network.** Communication dominates and throughput falls.\n- **Maximum parallelism degree by default.** Past the optimum, more GPUs is slower.\n- **Ignoring topology.** The strategy should follow the interconnect, not the model.\n- **Assuming it helps throughput as much as latency.** Its distinctive benefit is latency.\n- **Not profiling communication.** Time spent in collectives is invisible in GPU utilisation figures.\n- **Planning capacity in single GPUs.** The unit is the parallel group.\n\n## FAQ\n\n**How many GPUs should I use?**\nThe smallest number that fits the model and its KV cache, usually. Beyond that, additional degree buys latency at rising communication cost, and the curve flattens quickly. Measure rather than assume.\n\n**Does this apply to training as well?**\nYes, and training typically combines all three strategies — tensor within a node, pipeline across nodes, data parallel across replicas. The communication considerations are the same and the volumes are larger.\n\n**Do I need to implement this myself?**\nAlmost certainly not. Serving frameworks expose it as configuration. The value in understanding it is choosing the configuration and diagnosing why a setup underperforms.\n\n**Does quantisation reduce the need for it?**\nIt can remove the need entirely by making the model fit in fewer GPUs, which is frequently the cheaper answer — see [quantization explained](/blog/quantization-explained-2026).\n\n## Where to go next\n\nFor the across-node counterpart, read [pipeline parallelism](/blog/pipeline-parallelism-explained-2026). For the memory structure it splits, [paged attention](/blog/paged-attention-explained-2026), and for sizing the hardware, [GPU procurement](/blog/gpu-procurement-guide-2026).","hue":203,"metaTitle":"Tensor Parallelism: Splitting Layers Across GPUs","metaDescription":"How tensor parallelism divides individual layers across GPUs, why interconnect bandwidth limits it, and when pipeline parallelism is the better split.","keywords":["tensor parallelism","model parallelism gpu","nvlink bandwidth","all reduce communication","multi gpu inference","large model serving","distributed inference","gpu scaling 2026"],"tldr":["**Each GPU holds a slice of every layer** — so every device works on every token, unlike pipeline parallelism.","**It communicates on every layer** — which makes interconnect bandwidth the binding constraint.","**It is a within-node technique** — crossing a slower network boundary usually costs more than it gains.","**It reduces latency, not just capacity** — splitting the work genuinely speeds up a single request.","**Skip:** tensor parallelism across nodes connected by ordinary networking. Use pipeline parallelism at that boundary instead."],"readTime":"4 min read","wordCount":869},"related":[{"slug":"pipeline-parallelism-explained-2026","title":"Pipeline Parallelism: Splitting a Model Across Machines","excerpt":"Give each machine a contiguous block of layers and pass activations along the chain. It scales across ordinary networks, and it introduces idle time you have to engineer away.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":208},{"slug":"moe-routing-explained-2026","title":"MoE Routing Explained: How Experts Get Chosen","excerpt":"A mixture-of-experts model activates a fraction of its parameters per token. Here is how the router decides, why load balancing is hard, and what it means for serving.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":218},{"slug":"adapter-merging-explained-2026","title":"Adapter Merging: Folding a Fine-Tune Back Into the Model","excerpt":"A trained adapter can stay separate and swappable, or be merged into the base weights for faster inference. You cannot have both, and the choice shapes your serving architecture.","category":"ai","date":"January 7, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":221}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"spot-instances-for-training-2026","title":"Spot Instances for Training: Cheap Compute You Can Lose","excerpt":"Interruptible capacity costs a fraction of on-demand and can be reclaimed with little warning. Whether that is a bargain or a disaster depends entirely on your checkpointing.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":204},"older":{"slug":"add-stripe-subscriptions-to-ai-app-2026","title":"How to add Stripe subscriptions to your AI-generated app (2026 step-by-step)","excerpt":"Most Stripe tutorials stop at the payment form. This one covers the part that breaks in production: products, checkout sessions, webhooks, and the subscription state machine — for apps shipped from Cursor, v0, Bolt, or Lovable.","category":"guides","date":"January 1, 2026","author":"ByteLedger Team","readTime":"9 min read","hue":145}}