{"post":{"slug":"mixed-precision-training-2026","title":"Mixed Precision Training: Faster and Smaller, Carefully","excerpt":"Training in lower precision is faster and uses less memory. Doing it naively produces gradients that underflow to zero, which is why the technique is called mixed rather than low.","category":"ai","date":"January 5, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">L</span>ower-precision arithmetic is faster on modern accelerators and uses proportionally less memory. Training entirely in low precision, however, breaks — gradients are small numbers, and small numbers in a narrow-range format round to zero, so parts of the model simply stop learning.\n\nMixed precision resolves this by using low precision where it is safe and higher precision where it is not.\n\n## What changed in 2026\n\n- **Wider-range formats became the default.** Formats trading mantissa bits for exponent range removed the underflow problem and made loss scaling largely unnecessary.\n- **It stopped being optional.** Full-precision training became the exception, used only where a specific numerical issue demanded it.\n- **Lower precisions appeared for inference.** Eight-bit and below became common for serving while training stayed at higher precision.\n- **Framework defaults improved.** Automatic mixed precision with sensible operation-level policies reduced the manual work substantially.\n\n## What is mixed about it\n\nThe pattern: perform the expensive matrix operations in low precision, where the hardware is fastest, but keep a master copy of the weights and accumulate updates in higher precision.\n\nThat split matters because of a specific failure. Weight updates are typically much smaller than the weights themselves. Adding a tiny update to a large value in low precision can round to no change at all — the update vanishes, and training stalls even though gradients are being computed correctly. Keeping the master weights in higher precision preserves those small updates.\n\n| Component | Precision | Reason |\n| --- | --- | --- |\n| Matrix multiplications | Low | Speed; hardware is optimised for it |\n| Activations | Low | Memory; range is adequate |\n| Master weights | High | Small updates would vanish |\n| Optimiser state | High | Accumulates over many steps |\n| Loss and reductions | High | Summing many values loses precision |\n| Normalisation statistics | High | Sensitive to precision |\n\n## Underflow and loss scaling\n\nThe older narrow-range low-precision format had limited exponent range, and gradients frequently fall below its smallest representable value. They become exactly zero, and the corresponding parameters receive no update — silently, with no error.\n\nLoss scaling addresses this by multiplying the loss by a large factor before the backward pass, which scales all gradients up into representable range, then dividing by the same factor before the update. Dynamic loss scaling adjusts the factor automatically, backing off when it detects overflow.\n\nIt works and it is fiddly, which is why formats with wider exponent range — trading mantissa precision for range — became preferred. They represent small gradients directly, so loss scaling is unnecessary. Slightly less precision per value, considerably fewer numerical problems, and that trade has proven strongly favourable in practice.\n\nIf you are training with the older format and seeing parts of the model fail to learn, underflow is the first thing to check.\n\n## Diagnosing precision problems\n\nThe symptoms are distinctive once you know them.\n\n**Loss spikes** — sudden jumps in loss that recover — frequently indicate an overflow event in low precision. Loss scaling backing off looks like this.\n\n**NaN or infinity in the loss** means an overflow that was not caught. Once NaN enters the weights, the run is unrecoverable and must restart from a checkpoint.\n\n**Loss plateauing early** while gradients look reasonable can indicate underflow, with small gradients rounding to zero.\n\n**Divergence between a full-precision and mixed-precision run** on the same data and seed points at precision rather than at the recipe.\n\nThe practical response is to keep sensitive operations in higher precision — reductions, normalisations, and the loss computation — which framework policies generally do automatically. Overriding those defaults is where problems usually originate.\n\n## Common mistakes\n\n- **Full low precision including master weights.** Small updates vanish.\n- **Low-precision reductions.** Summing many values loses accuracy quickly.\n- **Disabling loss scaling on a narrow-range format.** Silent underflow.\n- **Blaming data for loss spikes.** Frequently a precision event.\n- **Continuing after a NaN.** The run is unrecoverable; restart from a checkpoint.\n- **Overriding framework precision policies without reason.** They encode hard-won defaults.\n- **Comparing runs at different precision.** Not directly comparable.\n\n## FAQ\n\n**Which format should I use?**\nThe wider-range one where your hardware supports it, since it avoids loss scaling entirely for a small precision cost. This has become the standard recommendation.\n\n**How much speedup should I expect?**\nMeaningful and hardware-dependent, since the gain comes from specialised units for low-precision matrix operations. Measure on your own setup rather than assuming a figure.\n\n**Does it affect final model quality?**\nProperly implemented, minimally. Improperly implemented — underflow, low-precision reductions — it can prevent convergence entirely, which is not a subtle quality difference.\n\n**Does it apply to inference?**\nInference typically goes further, to eight-bit and below, since there are no gradients to underflow. Different constraints, more aggressive compression — see [quantization explained](/blog/quantization-explained-2026).\n\n## Where to go next\n\nFor the other memory-reduction techniques this composes with, read [activation checkpointing](/blog/activation-checkpointing-2026) and [gradient accumulation](/blog/gradient-accumulation-explained-2026). For the inference-side equivalent, [quantization explained](/blog/quantization-explained-2026).","hue":217,"metaTitle":"Mixed Precision Training: Faster and Smaller, Carefully","metaDescription":"How mixed precision training speeds up and shrinks training, why gradient underflow requires loss scaling, and which operations must stay high precision.","keywords":["mixed precision training","bf16 vs fp16","loss scaling","gradient underflow","training speed optimization","numerical stability training","tensor cores","model training 2026"],"tldr":["**Compute in low precision, accumulate in high** — that split is what makes it stable.","**Small gradients underflow to zero in low precision** — loss scaling exists specifically to prevent it.","**Formats with wider range need no loss scaling** — which is why they largely displaced the older approach.","**Watch for loss spikes and NaNs** — the usual symptoms of a precision problem rather than a data problem.","**Skip:** full low precision for everything. Some operations genuinely need the range."],"readTime":"4 min read","wordCount":836},"related":[{"slug":"learning-rate-schedules-2026","title":"Learning Rate Schedules: The Hyperparameter That Matters Most","excerpt":"Of everything you can tune, the learning rate and how it changes over training has the largest effect on whether a run converges at all.","category":"ai","date":"January 18, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":202},{"slug":"activation-checkpointing-2026","title":"Activation Checkpointing: Recomputing to Save Memory","excerpt":"Training holds every intermediate activation for the backward pass. Discarding most of them and recomputing on demand trades compute for a large memory saving.","category":"ai","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":213},{"slug":"gradient-accumulation-explained-2026","title":"Gradient Accumulation: A Large Batch on Small Hardware","excerpt":"You need an effective batch size your GPU cannot hold. Accumulating gradients over several small passes before updating gets you the same result, more slowly.","category":"ai","date":"January 5, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":198}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"metadata-privacy-explained-2026","title":"Metadata Privacy: What Your Files Say Without the Contents","excerpt":"A photograph records where and when it was taken and on what device. A document records who wrote it and every edit. That information travels with the file unless you remove it.","category":"guides","date":"January 5, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":168},"older":{"slug":"reward-models-explained-2026","title":"Reward Models: Teaching a Model to Score Other Models","excerpt":"A reward model predicts which response a human would prefer. It is the scoring function behind RLHF, a useful evaluator on its own, and a reliable source of subtle failure.","category":"ai","date":"January 5, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":199}}