A model that does not fit in one GPU's memory has to be split across several. There are a few ways to cut it, and they have very different communication characteristics — which turns out to matter more than anything else about them.
Tensor parallelism makes the cut within each layer. Every GPU holds a slice of every layer's weights, every GPU participates in computing every token, and the partial results are combined after each layer.
What changed in 2026
- High-bandwidth interconnects spread. More clusters shipped with fast intra-node GPU links, widening where tensor parallelism is practical.
- Hybrid strategies became standard. Combining tensor parallelism within a node and pipeline parallelism across nodes settled as the default shape for very large models.
- Serving frameworks abstracted it. Configuring parallelism moved from bespoke code to a configuration parameter, which made getting it wrong easier and cheaper to fix.
- Communication became the documented bottleneck. Profiling showed collective operations dominating time in poorly-provisioned setups, which shifted attention to interconnect from raw compute.
How it differs from the alternatives
|
Tensor parallel |
Pipeline parallel |
Data parallel |
| Split by |
Within each layer |
Across layers |
Across batch |
| Each GPU holds |
A slice of everything |
Some complete layers |
A full copy |
| Communication |
Every layer |
Between stages |
Gradient sync |
| Latency for one request |
Reduced |
Not reduced |
Not reduced |
| Interconnect demand |
Very high |
Moderate |
Moderate |
| Fits a model too big for one GPU |
Yes |
Yes |
No |
The latency row is the reason to prefer tensor parallelism where you can afford it. Splitting the work of a single layer across four GPUs means four devices computing simultaneously on the same token, which genuinely reduces time to produce that token. Pipeline parallelism improves throughput by keeping stages busy across different requests; it does not make any individual request faster.
Communication is the constraint
The cost is unavoidable and structural. After each parallelised operation, partial results from every GPU must be combined before the next layer can proceed. That is a collective communication step, and it happens on every layer, for every token.
Which produces the defining property: tensor parallelism is bounded by interconnect bandwidth, not by compute.
Inside a node with a fast dedicated GPU interconnect, that communication is cheap enough to be worth it. Across nodes connected by ordinary networking, it is not — the collective operations dominate, and adding GPUs makes generation slower rather than faster. This is a genuinely counterintuitive outcome that surprises people scaling up.
Hence the standard hybrid: tensor parallelism within a node where the links are fast, pipeline parallelism across nodes where they are not. The parallelism strategy follows the hardware topology rather than the model architecture.
Degree matters too. Higher tensor parallelism means smaller per-GPU slices and more communication relative to compute. Past a point the communication overhead exceeds the compute saving, and the optimal degree is usually smaller than the number of GPUs available.
Interaction with the rest of serving
Tensor parallelism composes with the batching and memory techniques that govern throughput. The KV cache is also split across devices, which is part of how it relieves memory pressure — see paged attention for the cache side and continuous batching for the scheduling layer.
One practical consequence: a tensor-parallel deployment has a fixed GPU group. You cannot scale it by adding one GPU; you scale by adding another complete group. That granularity affects capacity planning and cost more than people expect — see AI capacity planning.
Common mistakes
- Using it across a slow network. Communication dominates and throughput falls.
- Maximum parallelism degree by default. Past the optimum, more GPUs is slower.
- Ignoring topology. The strategy should follow the interconnect, not the model.
- Assuming it helps throughput as much as latency. Its distinctive benefit is latency.
- Not profiling communication. Time spent in collectives is invisible in GPU utilisation figures.
- Planning capacity in single GPUs. The unit is the parallel group.
FAQ
How many GPUs should I use?
The smallest number that fits the model and its KV cache, usually. Beyond that, additional degree buys latency at rising communication cost, and the curve flattens quickly. Measure rather than assume.
Does this apply to training as well?
Yes, and training typically combines all three strategies — tensor within a node, pipeline across nodes, data parallel across replicas. The communication considerations are the same and the volumes are larger.
Do I need to implement this myself?
Almost certainly not. Serving frameworks expose it as configuration. The value in understanding it is choosing the configuration and diagnosing why a setup underperforms.
Does quantisation reduce the need for it?
It can remove the need entirely by making the model fit in fewer GPUs, which is frequently the cheaper answer — see quantization explained.
Where to go next
For the across-node counterpart, read pipeline parallelism. For the memory structure it splits, paged attention, and for sizing the hardware, GPU procurement.