Choosing hardware for inference is a different problem than choosing hardware for training, and treating them the same is the most common mistake teams make. Training rewards raw compute and interconnect bandwidth across thousands of chips running for weeks at a time. Inference rewards memory bandwidth, KV-cache capacity, and low latency on requests that arrive one at a time or in small batches, often for years after training ends. For most teams, GPUs remain the safer default because of software maturity and portability; TPUs and purpose-built inference chips win on raw cost per token once traffic is large, stable, and running on a well-supported architecture. The decision increasingly gets made at the serving-framework level rather than the chip level, since frameworks like vLLM now abstract away much of the difference — but the underlying economics of memory bandwidth and utilization still favor one option over another for any given traffic pattern.
What changed in 2026
- Inference-specific chip variants became the norm. Google's TPU v6e and AWS Trainium2/Inferentia2 are tuned specifically for serving, distinct from the chips those vendors sell for training runs.
- FP8 and FP4 quantization moved from experimental to default, cutting memory footprint and lifting effective throughput on every major chip family without a meaningful accuracy hit for most workloads.
- Disaggregated prefill and decode became a standard serving architecture, letting teams run the compute-heavy prompt phase and the bandwidth-heavy generation phase on separate hardware pools.
- Open serving stacks added multi-backend support. vLLM, TensorRT-LLM, and SGLang now target multiple chip families, so switching hardware no longer means rewriting the serving layer.
- Cost transparency improved industry-wide. Cloud providers and serving vendors now publish more granular cost-per-million-token figures by hardware tier, making an apples-to-apples comparison far easier than it was even a year earlier.
The hardware landscape
| Hardware |
Strength |
Tradeoff |
| NVIDIA H100 / H200 |
Broadest framework support, mature tooling |
Highest on-demand price per chip-hour |
| NVIDIA B200 / GB200 |
Large jump in memory bandwidth and FP4 throughput |
Newer, capacity constrained through 2026 |
| Google TPU v5e / v6e |
Strong cost per token at scale on supported models |
Effectively single-cloud, narrower framework surface |
| AWS Inferentia2 / Trainium2 |
Lower cost per token on AWS for supported model families |
Requires compiling through the Neuron SDK |
| Groq LPU / custom ASICs |
Very low, deterministic latency on fixed architectures |
Least flexible, expensive to re-target to a new model |
Total cost of ownership rarely shows up on a spec sheet. Porting a serving stack to a new chip family costs engineering time, and that cost is real even when the sticker price per hour looks attractive. Factor in egress costs if your data and users are not already inside the same cloud as the chip, and budget for a re-benchmarking pass every time a new hardware generation ships, since the ranking in the table above is not static.
How to actually choose
- Start from your cloud commitment. Multi-cloud or on-prem requirements rule out TPUs and Inferentia as a realistic default.
- Check model architecture support. A brand-new architecture may only have a mature compiled path on GPUs so far.
- Benchmark your own traffic shape. Test at your real batch size, sequence length, and concurrency, not a vendor's demo numbers.
- Separate prefill and decode once traffic is large enough to justify the added operational complexity.
- Re-evaluate at scale thresholds. The cheapest chip at low volume is rarely the cheapest chip at high, stable volume.
Common mistakes
- Reusing training benchmarks for a serving decision. Training throughput numbers say almost nothing about tail latency under real concurrent traffic.
- Ignoring memory bandwidth in favor of FLOPs. A chip with more raw compute but less bandwidth can still lose on inference throughput.
- Committing to single-cloud hardware before traffic is stable. Locking into TPU or Inferentia during an experimentation phase adds porting risk for an uncertain payoff.
- Skipping quantization evaluation. FP8/FP4 support varies enough by chip and framework that it can flip a cost comparison entirely.
FAQ
Is a TPU always cheaper than a GPU for inference?
Only at scale, on a supported architecture, within its native cloud. At low or bursty volume, the porting cost usually is not worth it.
Do I need different hardware for training and inference?
Not necessarily, but many teams split them, since the cost profile that wins for training rarely wins for serving.
What matters more, FLOPs or memory bandwidth?
For most transformer inference at typical batch sizes, memory bandwidth and KV-cache capacity matter more than peak compute.
Is a purpose-built inference chip worth it for a startup?
Rarely at first. The payoff shows up once traffic is stable, high-volume, and concentrated on one well-understood model family.
Can I switch hardware after launch without a rewrite?
Usually yes if you built on an open serving stack like vLLM or TensorRT-LLM, since the model-serving interface stays the same even when the backend chip changes. A serving stack tied tightly to one vendor's SDK makes switching far more expensive.
Where to go next
Pair this with inference optimization techniques for 2026 once hardware is chosen, see how streaming vs batch processing frames a related throughput-versus-latency tradeoff, and review sync vs async for how concurrency affects the serving layer above the hardware.