Lower-precision arithmetic is faster on modern accelerators and uses proportionally less memory. Training entirely in low precision, however, breaks — gradients are small numbers, and small numbers in a narrow-range format round to zero, so parts of the model simply stop learning.
Mixed precision resolves this by using low precision where it is safe and higher precision where it is not.
What changed in 2026
- Wider-range formats became the default. Formats trading mantissa bits for exponent range removed the underflow problem and made loss scaling largely unnecessary.
- It stopped being optional. Full-precision training became the exception, used only where a specific numerical issue demanded it.
- Lower precisions appeared for inference. Eight-bit and below became common for serving while training stayed at higher precision.
- Framework defaults improved. Automatic mixed precision with sensible operation-level policies reduced the manual work substantially.
What is mixed about it
The pattern: perform the expensive matrix operations in low precision, where the hardware is fastest, but keep a master copy of the weights and accumulate updates in higher precision.
That split matters because of a specific failure. Weight updates are typically much smaller than the weights themselves. Adding a tiny update to a large value in low precision can round to no change at all — the update vanishes, and training stalls even though gradients are being computed correctly. Keeping the master weights in higher precision preserves those small updates.
| Component |
Precision |
Reason |
| Matrix multiplications |
Low |
Speed; hardware is optimised for it |
| Activations |
Low |
Memory; range is adequate |
| Master weights |
High |
Small updates would vanish |
| Optimiser state |
High |
Accumulates over many steps |
| Loss and reductions |
High |
Summing many values loses precision |
| Normalisation statistics |
High |
Sensitive to precision |
Underflow and loss scaling
The older narrow-range low-precision format had limited exponent range, and gradients frequently fall below its smallest representable value. They become exactly zero, and the corresponding parameters receive no update — silently, with no error.
Loss scaling addresses this by multiplying the loss by a large factor before the backward pass, which scales all gradients up into representable range, then dividing by the same factor before the update. Dynamic loss scaling adjusts the factor automatically, backing off when it detects overflow.
It works and it is fiddly, which is why formats with wider exponent range — trading mantissa precision for range — became preferred. They represent small gradients directly, so loss scaling is unnecessary. Slightly less precision per value, considerably fewer numerical problems, and that trade has proven strongly favourable in practice.
If you are training with the older format and seeing parts of the model fail to learn, underflow is the first thing to check.
Diagnosing precision problems
The symptoms are distinctive once you know them.
Loss spikes — sudden jumps in loss that recover — frequently indicate an overflow event in low precision. Loss scaling backing off looks like this.
NaN or infinity in the loss means an overflow that was not caught. Once NaN enters the weights, the run is unrecoverable and must restart from a checkpoint.
Loss plateauing early while gradients look reasonable can indicate underflow, with small gradients rounding to zero.
Divergence between a full-precision and mixed-precision run on the same data and seed points at precision rather than at the recipe.
The practical response is to keep sensitive operations in higher precision — reductions, normalisations, and the loss computation — which framework policies generally do automatically. Overriding those defaults is where problems usually originate.
Common mistakes
- Full low precision including master weights. Small updates vanish.
- Low-precision reductions. Summing many values loses accuracy quickly.
- Disabling loss scaling on a narrow-range format. Silent underflow.
- Blaming data for loss spikes. Frequently a precision event.
- Continuing after a NaN. The run is unrecoverable; restart from a checkpoint.
- Overriding framework precision policies without reason. They encode hard-won defaults.
- Comparing runs at different precision. Not directly comparable.
FAQ
Which format should I use?
The wider-range one where your hardware supports it, since it avoids loss scaling entirely for a small precision cost. This has become the standard recommendation.
How much speedup should I expect?
Meaningful and hardware-dependent, since the gain comes from specialised units for low-precision matrix operations. Measure on your own setup rather than assuming a figure.
Does it affect final model quality?
Properly implemented, minimally. Improperly implemented — underflow, low-precision reductions — it can prevent convergence entirely, which is not a subtle quality difference.
Does it apply to inference?
Inference typically goes further, to eight-bit and below, since there are no gradients to underflow. Different constraints, more aggressive compression — see quantization explained.
Where to go next
For the other memory-reduction techniques this composes with, read activation checkpointing and gradient accumulation. For the inference-side equivalent, quantization explained.