The intuition that a faster chip runs a model faster is wrong for most AI inference. Generating a token requires reading the model weights out of memory, and on a large model that read is enormous. The arithmetic itself finishes quickly and then the compute units wait. Throughput is set by how fast bytes move, not by how fast math happens.
High-bandwidth memory exists to narrow that gap, and HBM4 is the current step.
What changed in 2026
- The interface got wider. HBM4 moves to a substantially wider per-stack interface than HBM3E, which is the primary source of its bandwidth gain rather than a large clock increase.
- Capacity per stack increased. Taller stacks and denser dies mean more model fits per accelerator, reducing the number of devices a given model must be split across.
- Custom base dies became a differentiator. Rather than a standard interposer layer, vendors began tailoring the logic die under the memory stack to specific accelerator designs.
- Memory became the supply bottleneck. Accelerator shipment forecasts started being quoted against HBM production capacity, an inversion of the usual relationship between logic and memory.
Why bandwidth is the number that matters
| Workload |
Bound by |
Improved most by |
| Large model token generation |
Memory bandwidth |
Faster memory |
| Prompt processing and prefill |
Compute |
More arithmetic throughput |
| Training with large batches |
Mixed; interconnect matters too |
Bandwidth plus fabric |
| Small model inference |
Often compute or overhead |
Faster clocks, lower latency |
| Batch inference at high concurrency |
Bandwidth, then capacity |
Faster and larger memory |
The practical implication for anyone comparing hardware is that a specification sheet leading with peak arithmetic performance is telling you about the prefill phase and almost nothing about generation speed. Bandwidth per unit of compute is the more predictive ratio, and it is the number vendors are less eager to headline.
Capacity matters for a different reason. When a model does not fit in one device's memory, it must be split across devices, and the communication between them becomes a new bottleneck — which is why interconnect quality shows up so prominently when leasing AI compute. More capacity per device means fewer devices and less of that traffic.
What this means downstream
For anyone buying inference capacity rather than hardware, memory economics show up as pricing. Providers with newer memory serve more tokens per second per device, which is why per-token prices on the same nominal model can differ between providers.
For anyone running models locally, the same rule applies at smaller scale: memory bandwidth on a consumer machine is what determines local generation speed, far more than the marketing number on the accelerator. This is the mechanism behind why unified-memory laptops sometimes outperform discrete cards with more raw compute on large-model inference — and it is covered from the device side in AI PC TOPS explained.
Common mistakes
- Comparing accelerators on peak FLOPS. It predicts prefill speed, not generation speed, and generation is what users feel.
- Ignoring capacity when choosing a model size. A model that just barely does not fit forces sharding and a large performance cliff.
- Assuming memory improvements are incremental. Bandwidth is the direct multiplier on token throughput; a generational jump in memory is a generational jump in inference speed.
- Overlooking power. High-bandwidth memory is a significant share of accelerator power draw, which affects density and cooling in a datacenter.
- Treating memory supply as guaranteed. HBM capacity is contracted far in advance and is a real constraint on hardware availability.
FAQ
Is HBM4 available in shipping products?
It is entering the market in next-generation accelerator designs. Availability follows the accelerator launch cycles rather than being a separate consumer purchase.
Does more bandwidth help training as much as inference?
It helps, but training is more sensitive to interconnect between devices and to compute throughput. Inference generation is the case where bandwidth dominates most clearly.
Why not just use ordinary DRAM?
Bandwidth. Standard memory interfaces cannot supply data anywhere near fast enough for a large accelerator, which is the entire reason stacked memory adjacent to the die exists.
Does this affect consumer hardware?
Indirectly. Consumer accelerators use different memory types, but the same principle applies — bandwidth gates large-model inference on a desktop just as it does in a datacenter.
Where to go next
For the accelerator side of the story, read AI chips explained. For how memory constraints translate into what you pay, see AI compute leasing explained, and for advanced packaging, glass substrate packaging.