Picking a quantization format is a different question from understanding what quantization does. Once you know you want a smaller, faster model, the practical problem is which packaged format to download, and the honest answer depends entirely on where the model will run. A GGUF file that flies on a laptop through llama.cpp will not even load in a GPU serving stack expecting AWQ, and the reverse is just as true. This is a field guide to picking correctly the first time. For the underlying mechanics of what quantization actually trades away, see quantization explained.
What changed in 2026
- FP8 went from a data-center curiosity to a realistic option for teams with recent NVIDIA hardware, offering noticeably better quality retention than 4-bit integer formats at a similar speed benefit.
- GGUF's surrounding tooling got significantly more polished, cementing it as the default for anyone running models locally rather than on a server.
- AWQ largely overtook older GPTQ checkpoints as the go-to activation-aware format for GPU serving, though GPTQ remains widely available and still well supported.
The formats, briefly
GGUF is the format behind llama.cpp, Ollama, and LM Studio, built for running models on regular consumer hardware, including CPU-only setups, with a wide range of bit widths in a single file type. GPTQ is a post-training method that calibrates on a small sample dataset, historically strong on GPU inference servers. AWQ is a newer activation-aware method that tends to preserve quality better than GPTQ at the same bit width, and has become the more common choice in serving stacks like vLLM. EXL2 targets the ExLlama runtime specifically, allowing mixed bit-widths across a single model. bitsandbytes quantizes on the fly inside a Transformers pipeline rather than shipping a pre-quantized file, convenient for experimentation but generally slower than a purpose-built format for production. ONNX quantization targets cross-platform and edge deployment, including mobile and browser runtimes, where GGUF and GPTQ do not apply at all.
Formats compared by use case
| Format |
Runtime |
Best for |
Watch out for |
| GGUF |
llama.cpp, Ollama, LM Studio |
Local / CPU / consumer GPU inference |
Not built for high-throughput server serving |
| GPTQ |
vLLM, Transformers, ExLlama |
GPU serving, wide tooling support |
Being edged out by AWQ for new deployments |
| AWQ |
vLLM, TGI |
GPU serving with better quality retention |
Narrower tool support than GPTQ historically |
| EXL2 |
ExLlama / ExLlamaV2 |
Fine-grained size and quality tradeoffs |
Smaller ecosystem, fewer pre-quantized checkpoints |
| FP8 |
vLLM, TensorRT-LLM on Hopper-class or newer |
Best quality retention at speed, if hardware supports it |
Requires recent NVIDIA GPUs |
| ONNX (quantized) |
ONNX Runtime, mobile / edge |
Cross-platform, browser, mobile deployment |
Conversion step adds friction; not LLM-specific tooling |
| bitsandbytes |
Transformers (Python) |
Fast experimentation, fine-tuning |
Slower inference than dedicated serving formats |
How to actually choose
Start from the runtime, not the format. Running a model locally on a laptop or a machine without a serious GPU: GGUF through Ollama or llama.cpp is the practical default, since the tooling is the most mature and forgiving. Serving a model to multiple users through an API: AWQ or FP8, hardware permitting, through vLLM is the current strong default for GPU throughput. Deploying to mobile or a browser: ONNX is close to the only option. Chasing the best format in the abstract, disconnected from where the model actually runs, is how people end up with three half-working conversion scripts and no working deployment.
Where quality actually goes
Every format loses some quality relative to the full-precision original, and the loss is not evenly distributed — it shows up disproportionately on tasks requiring precise reasoning, arithmetic, or long-context recall, and much less on conversational or summarization tasks. Test on your actual use case before committing to a bit width, not on a generic benchmark. A format that looks fine on a leaderboard can still degrade the specific task you care about.
FAQ
Which quantization format is fastest?
It depends more on hardware match than the format itself. GGUF on CPU is fast for CPU-only setups; AWQ or FP8 on a matching GPU will outperform GGUF running on that same GPU, because they are built for that serving path specifically.
Can I convert between quantization formats?
Sometimes, with varying quality loss and effort — GPTQ-to-AWQ conversion tools exist, for example. It is usually easier and safer to download a pre-quantized checkpoint in the format you need than to convert one yourself.
Is FP8 always better than 4-bit integer formats?
Generally yes on quality, but it needs recent NVIDIA hardware and is not universally supported across every serving stack yet. Check hardware compatibility before assuming it is available to you.
Do I need to understand quantization theory to pick a format?
Not deeply. Knowing the runtime you are targeting and matching it to the table above covers most real decisions. The theory matters more once you are troubleshooting an unexpected quality drop.
Where to go next