Quantization is how a model that nominally needs a data-center GPU ends up running on a laptop. The idea is straightforward: model weights are just numbers, and you do not always need full precision to store them. By representing each weight in fewer bits, you shrink the model, cut memory bandwidth, and speed up inference — often with surprisingly little loss in quality. The art is knowing how far you can push it before the model starts to degrade.
What changed in 2026
- 4-bit became the default for local inference. Running capable open-weight models at 4-bit precision is now routine, making strong models accessible on consumer hardware.
- Quantization formats consolidated. A handful of well-supported schemes now dominate tooling, so the earlier fragmentation of incompatible formats has eased.
- Low-bit research advanced. Methods for pushing below 4-bit improved, though the quality cliff at very low precision remains real for demanding tasks.
How quantization works
A model normally stores weights as 16-bit floating-point numbers. Quantization maps those values onto a smaller set of levels represented with fewer bits — 8-bit integers, 4-bit, or lower. Because there are fewer possible values, each weight takes less space, and the arithmetic can run faster on hardware optimized for low-precision math.
The cost is rounding error: each weight is nudged to the nearest representable level. Good quantization methods minimize the damage by handling outlier weights carefully and calibrating on sample data so the important values keep their fidelity.
Precision levels compared
| Precision |
Memory vs 16-bit |
Typical quality |
Common use |
| 16-bit (FP16/BF16) |
Baseline |
Full |
Training, high-fidelity serving |
| 8-bit |
~2x smaller |
Near-full |
Safe general deployment |
| 4-bit |
~4x smaller |
Slight loss |
Local and cost-sensitive inference |
| 3-bit and below |
Smaller still |
Noticeable loss |
Extreme memory constraints |
Four-bit is where most practitioners land: the memory savings are large and the quality drop is small enough to be acceptable for many tasks. Below that, the tradeoff sharpens quickly.
Two ways to quantize
- Post-training quantization (PTQ). Take a trained model and quantize it directly, usually with a small calibration dataset to set scaling. Fast and cheap, and good enough down to 4-bit for most models.
- Quantization-aware training (QAT). Simulate low precision during training or fine-tuning so the model learns to be robust to it. More expensive, but it preserves quality at aggressive bit widths where PTQ falters.
For most teams, PTQ is the starting point; QAT is worth the effort only when you need very low precision without losing accuracy.
Where quantization fits
Quantization is a deployment tool, not a training shortcut. It pairs naturally with other efficiency techniques: it makes large mixture of experts models fit in memory, and it composes with adapter-based tuning — see LoRA vs full fine-tuning, where quantized base weights plus a small trainable adapter is a popular, memory-frugal combination.
FAQ
Does quantization make a model dumber?
It introduces small rounding errors, so there is some quality loss, but at 8-bit and often 4-bit the drop is minor for most tasks. The degradation becomes serious only at very low bit widths or on precision-sensitive workloads.
Does quantization speed up inference or just save memory?
Both, usually. Lower precision reduces the memory the model occupies and the bandwidth needed to move weights, and hardware can run low-precision math faster, so latency and throughput typically improve.
Can I quantize during training?
Quantization-aware training simulates low precision while training so the model adapts to it, but standard quantization targets an already-trained model for inference. Training itself is generally done in higher precision.
Which quantization format should I use?
Pick whatever your serving stack supports best, and prefer widely adopted 4-bit and 8-bit schemes. Benchmark the quantized model on your own tasks rather than trusting a generic quality claim.
Where to go next