Picking the right GPU for AI workloads in 2026 comes down to three questions: how much VRAM do you need, how often do you need it, and whether you are inferencing or training. Everything else — compute TFLOPS, memory bandwidth, tensor core generation — follows from those answers.
What changed in 2026
- RTX 5090 launched with 32 GB GDDR7, making it the first consumer card that comfortably runs 22B parameter models without quantisation.
- H200 SXM is the new data centre baseline, replacing H100 for new cluster deployments with 141 GB HBM3e and 3.35 TB/s bandwidth.
- AMD ROCm 7.0 became genuinely usable. RX 9900 XTX competes with RTX 5080 for inference; PyTorch and vLLM support is stable.
- Mac M4 Ultra (192 GB unified memory) can run a 70B model at reasonable speed — a legitimate option for on-device inference without NVIDIA.
GPU comparison: consumer cards
| GPU |
VRAM |
Bandwidth |
Max model (Q4) |
Price (approx) |
| RTX 4060 Ti |
16 GB |
288 GB/s |
7B |
~$500 |
| RTX 4090 |
24 GB |
1,008 GB/s |
13B |
~$1,600 |
| RTX 5080 |
16 GB |
960 GB/s |
7B |
~$1,000 |
| RTX 5090 |
32 GB |
1,792 GB/s |
22B |
~$2,000 |
| RX 9900 XTX |
32 GB |
1,280 GB/s |
22B |
~$1,000 |
GPU comparison: data centre
| GPU |
VRAM |
Bandwidth |
Use case |
Cloud spot (~) |
| A10G |
24 GB |
600 GB/s |
Inference, fine-tuning 7B |
$0.80/hr |
| A100 40 GB |
40 GB |
2 TB/s |
Training up to 30B |
$1.50/hr |
| A100 80 GB |
80 GB |
2 TB/s |
Training 30B–70B |
$2.50/hr |
| H100 SXM |
80 GB |
3.35 TB/s |
Large model training |
$3–5/hr |
| H200 SXM |
141 GB |
4.8 TB/s |
70B+ training/inference |
$5–8/hr |
How to pick by workload
Local inference (personal/dev):
RTX 4090 or RTX 5090. The 24/32 GB VRAM covers 90% of open-weight models. RTX 5090 is the 2026 recommendation if budget allows.
Fine-tuning a 7B model (LoRA):
Any 16 GB+ card runs QLoRA for a 7B model. A10G or RTX 4090 in the cloud if you do not own hardware.
Fine-tuning a 70B model:
You need 2× A100 80 GB minimum. Cloud is almost always cheaper than ownership at this tier.
Production inference at scale:
vLLM on A10G spot instances for 7B models. For 70B at high throughput, H100 is the right tool.
Air-gapped / on-device:
Mac M4 Ultra (192 GB) runs 70B models locally. RTX 5090 workstation is the NVIDIA equivalent for smaller models.
VRAM is the binding constraint
For inference, the model weights must fit in VRAM. At 4-bit quantisation:
- 7B model ≈ 4 GB weights + 4 GB KV cache = ~8 GB minimum, 12 GB comfortable
- 13B model ≈ 7 GB weights + headroom = 16 GB minimum
- 22B model ≈ 12 GB weights = 16 GB minimum, 24 GB comfortable
- 70B model ≈ 35 GB weights = 40+ GB, comfortably 80 GB
For training, multiply by ~3–4× for gradients and optimiser state (or use LoRA to reduce this).
How to start
- Decide: inference only or training? Inference needs less VRAM; training needs more.
- Pick the smallest model that meets quality requirements — a 7B model on a 16 GB GPU may beat a 70B model on $10k hardware for your use case.
- Cloud first unless your workload is 24/7 — ownership breaks even only at high utilisation.
- Benchmark on your actual workload — token/sec on your prompt distribution, not synthetic benchmarks.
Common mistakes
Buying based on TFLOPS alone. Memory bandwidth determines LLM inference speed more than compute. The RTX 5090 outperforms the 4090 not because of more CUDA cores but because of higher bandwidth.
Undersizing VRAM. 8 GB cards run heavily quantised 3B models with no headroom. This is experimentation hardware, not production.
Renting on-demand instead of spot. For batch or dev workloads, spot/preemptible instances are 60–80% cheaper.
Ignoring AMD. ROCm 7.0 with RX 9900 XTX is a serious option for inference at half the NVIDIA cost if you are willing to test compatibility.
What to skip
- RTX 4060 (8 GB) for anything beyond running quantised 3B models and testing the local LLM workflow.
- Dual-socket CPU inference rigs — llama.cpp on CPU is 20× slower than a mid-range GPU. Spend the money on VRAM, not RAM.
- Used data centre GPUs without due diligence — V100s and P100s predate modern quantisation kernels and lack bfloat16 support.
FAQ
Is an RTX 5090 better than renting cloud GPUs?
It depends on utilisation. At 8+ hours per day of active use, ownership pays back in ~18 months. Below that, cloud is cheaper.
Can I use an Intel Arc GPU for AI?
Arc A-series GPUs have limited PyTorch support. The B-series (2025) is better but still behind NVIDIA/AMD for LLM inference.
What is the minimum useful GPU in 2026?
RTX 4060 Ti (16 GB) is the practical floor for useful local model inference. Below that, you are limited to heavily quantised 3B models.
Does more VRAM always help?
For inference, once the model fits, more VRAM allows larger batch sizes and context length, which improves throughput but not individual request latency significantly.
Where to go next