Quantized models store their weights using fewer bits than the precision they were originally trained in, trading numeric detail for a smaller footprint and faster inference. A model trained in 16-bit precision and quantized to 4-bit needs roughly a quarter of the memory, which is why quantization is what lets large models run on consumer GPUs, laptops, and phones at all. The catch is that rounding numbers loses information, and that loss shows up unevenly: some tasks barely notice, while others, particularly multi-step reasoning and precise numeric or code generation, can degrade noticeably once precision drops too far.
How it works
Every weight in a neural network is a number, and in full precision that number is stored with enough bits to represent very fine distinctions. Quantization maps that wide range of possible values onto a much smaller set of representable values, typically using a scale and offset computed per layer or per group of weights, then rounds each original weight to its nearest representable point. The model still runs the same architecture and the same math; it is just doing that math with coarser numbers, and reading and moving those numbers around costs less memory bandwidth, which is often the actual bottleneck in inference speed.
Two quantization strategies dominate in practice. Post-training quantization takes an already-trained model and converts it directly, which is fast and requires no retraining but tends to lose a bit more quality at very low bit widths. Quantization-aware training instead simulates the rounding during training itself, so the model learns weights that tolerate quantization better, at the cost of extra training time and compute. Most publicly available quantized model files use post-training quantization because it is far cheaper to produce, and modern variants such as GPTQ and AWQ improve on naive rounding by identifying and protecting the small number of outlier weights that matter disproportionately to output quality.
The tradeoff triangle
| Precision |
Relative memory vs fp16 |
Typical inference speed |
Typical quality impact |
| fp16 / bf16 (baseline) |
100 percent |
Baseline |
None, reference quality |
| int8 |
Roughly 50 percent |
Faster, memory-bandwidth limited |
Minimal on most tasks |
| int4 (GPTQ, AWQ, GGUF Q4) |
Roughly 25 percent |
Fastest, smallest footprint |
Noticeable on reasoning-heavy tasks |
| int2-3 (aggressive) |
Roughly 12-19 percent |
Fastest but often unstable |
Significant, task-dependent |
No single row is universally "correct." A customer support chatbot answering short factual questions can often run at int4 with no perceptible quality loss, while a coding assistant doing multi-step logic may show real regressions at the same bit width. This is why the tradeoff is best understood as a triangle: pick two priorities, such as size and speed, and quality is what typically absorbs the difference.
Where quantization breaks down
Quantization damage concentrates in specific places rather than spreading evenly across a model. Outlier weights, a small number of unusually large values that carry disproportionate signal, are the main source of error if rounded carelessly, which is why GPTQ, AWQ, and similar schemes specifically identify and protect them. Long chains of reasoning compound small errors at each step, so heavily quantized models tend to lose more accuracy on multi-step math or logic than on single-turn classification. Rare or unusual inputs, including uncommon languages, dense technical notation, or long numeric sequences, are also more likely to expose the model producing the wrong token where a full-precision model would not have.
Common mistakes
- Assuming a benchmark score at one bit width generalizes to your workload. Public leaderboards test general tasks; a coding-heavy or reasoning-heavy workload can show a bigger quality gap than the headline number suggests.
- Quantizing too aggressively for the available hardware headroom. Choosing int3 to save memory that was not actually needed adds quality risk with no real benefit; match the bit width to the actual memory constraint.
- Ignoring which layers were protected. Not all int4 files are equal; two models labeled "4-bit" can differ meaningfully depending on whether outlier weights were protected during conversion.
- Skipping a side-by-side comparison against the full-precision model. Without running the same prompts through both versions, subtle regressions in reasoning or formatting can go unnoticed until they reach production.
FAQ
Does quantization make a model dumber?
It can, but the effect is task-dependent and usually modest at int8 and moderate at int4 with a well-implemented scheme. The safest approach is testing your own prompts rather than relying on a general reputation.
What is the difference between quantization and pruning?
Quantization reduces the precision of existing weights; pruning removes weights or connections entirely. They target the same goal, a smaller and faster model, through different mechanisms, and are sometimes combined.
Is int4 quantization safe for production use?
Often yes for short-form generation, classification, and retrieval-style tasks. It deserves more caution for long reasoning chains, code generation, or any task where a single wrong token has outsized consequences.
How does this relate to fine-tuning a model?
They are independent but combinable. QLoRA specifically fine-tunes a quantized base model, which is covered in our guide to parameter-efficient fine-tuning.
Where to go next
For the hands-on side of this topic, see our step-by-step guide to running quantized AI models, plus our explainers on parameter-efficient fine-tuning and open-source vs closed-source LLMs.