The specification sheet leads with compute throughput, so that is what gets compared. For inference workloads it is frequently the least relevant number on the page.
What decides whether a GPU works for you is whether your model and its cache fit in memory, and how fast that memory can be read. Compute matters, and it is rarely the thing you run out of first.
What changed in 2026
- Memory capacity became the headline selection criterion. As models and contexts grew, fitting the workload displaced raw speed as the primary question.
- The rental market matured. Committed-capacity and spot options made owning hardware a narrower proposition — see reserved GPU capacity.
- Interconnect specifications got scrutinised. Teams learned that multi-GPU strategy depends on link bandwidth, not GPU count.
- Older generations stayed viable. For inference at moderate scale, previous-generation hardware with adequate memory remained cost-effective.
What to compare
| Specification |
Matters for |
Priority for inference |
| Memory capacity |
Whether the workload fits |
Highest |
| Memory bandwidth |
Generation speed |
High |
| Interconnect bandwidth |
Multi-GPU strategy |
High if scaling out |
| Compute throughput |
Prefill speed, training |
Moderate |
| Power and cooling |
Operating cost, density |
Moderate |
Memory capacity first. Model weights plus KV cache for your target concurrency plus overhead must fit. Under-provision and you either cannot run the model or run it at concurrency too low to be economical. This is a hard constraint, not a performance gradient.
Bandwidth second. Token generation reads the weights repeatedly, so bandwidth sets the ceiling on generation speed more directly than compute does — see GPU utilisation.
Interconnect third, and it is a gate rather than a gradient. Fast intra-node links make tensor parallelism viable; slow ones do not. If you will split a model across devices, this determines which strategies are available at all.
Compute last for inference. It matters for prefill and for training, and it is rarely the first thing you exhaust when serving.
Sizing the memory requirement
Work it out rather than guessing, because the cache term surprises people.
Start with model weights at your serving precision. Quantisation reduces this substantially and is frequently the cheapest way to fit a model on smaller hardware — see quantization explained.
Then add KV cache, which scales with context length times concurrent sequences. This is the term that dominates at long contexts, and it is where teams under-provision because they sized for the weights alone.
Then leave headroom for activations and fragmentation.
The useful exercise is computing the concurrency your memory supports at your typical context length, then checking whether that concurrency meets your throughput requirement. If it does not, you need more memory rather than more compute — a conclusion that is only visible if you do the arithmetic.
Rent before you buy
Owned hardware is cheaper per hour at high utilisation and worse in every other respect: capital outlay, a fixed capacity that cannot follow demand, and an obsolescence curve that has been steep.
The pattern that works: rent while your workload is uncertain, measure actual sustained utilisation over months, and consider owning only the stable baseline. Utilisation on owned hardware almost always comes in below forecast, because forecasts are built from peaks.
Where you do commit, committing to a baseline and renting the peaks beats owning for peak. Idle owned capacity is pure loss; overflow rental is a variable cost you only pay when you need it.
Common mistakes
- Comparing on compute throughput. Rarely the binding constraint for inference.
- Sizing memory for weights only. The cache term dominates at long contexts.
- Ignoring interconnect. It determines which parallelism strategies work.
- Buying for peak demand. Idle capacity is unrecoverable.
- Assuming newest is required. Previous generations with adequate memory are often better value.
- Not measuring utilisation before buying. Forecasts overstate it consistently.
- Overlooking power and cooling. Real constraints in a rack, and real cost.
FAQ
How much memory do I need?
Compute it: weights at serving precision, plus cache for your concurrency at your context length, plus headroom. There is no rule of thumb because the cache term depends entirely on your workload shape.
Is it cheaper to own?
Only at high sustained utilisation, and only if you measured that utilisation rather than forecasting it. Most teams over-estimate.
Should I buy consumer or datacentre hardware?
Consumer cards can be cost-effective for development and small-scale inference. Datacentre parts bring more memory, better interconnect, and licensing terms that matter for production deployment.
Does quantisation change what I should buy?
Substantially — it can move a workload from requiring several GPUs to one. Establish your serving precision before selecting hardware, not after.
Where to go next
For the throughput metrics that should guide selection, read GPU utilisation. For committed capacity options, reserved GPU capacity and spot instances for training.