If you could tune exactly one thing about a training run, it would be the learning rate. Too large and the run diverges or oscillates without settling. Too small and it converges slowly to a worse result, wasting compute for a lesser model.
And a single value is rarely right for the whole run, because what the model needs early differs from what it needs late.
What changed in 2026
- Warmup became universal. Essentially all large-model training uses it, and its absence is treated as a bug rather than a choice.
- Decay shapes converged. A small number of well-understood shapes displaced the proliferation of exotic schedules.
- Batch-size scaling rules got clearer. Adjusting the peak rate when changing batch size became routine rather than rediscovered.
- Fine-tuning rates settled far lower. The gap between pretraining and fine-tuning rates became well-established practice.
Warmup, and why it exists
At the start of training, weights are near-random and gradients are large and poorly conditioned. Applying full-size updates immediately can push the model into a state it never recovers from — a divergence in the first few hundred steps.
Warmup starts the learning rate near zero and raises it to the peak over some number of steps. The model takes small, cautious updates while things are chaotic, and full-size ones once the gradients are more meaningful.
The interaction with adaptive optimisers matters too: those maintain running estimates of gradient statistics, and early on those estimates are based on very few samples and are unreliable. Warmup gives them time to stabilise before their scaling has much effect.
Decay, and the shapes that work
A rate that makes good progress early is too large late. Near a good solution, large steps overshoot and the loss oscillates instead of settling.
| Shape |
Behaviour |
Common use |
| Constant |
No decay |
Rarely optimal; simple baseline |
| Linear decay |
Steady decline to zero |
Simple, works well |
| Cosine decay |
Slow, then fast, then slow |
The common default |
| Step decay |
Drops at fixed points |
Older practice; abrupt |
| Constant then decay |
Long plateau, late decay |
Useful when total steps are unknown |
Cosine decay to a small final value is the widespread default and performs well across a wide range of settings. The bottom row is worth knowing: if you do not know in advance how long a run will last, a schedule requiring a known endpoint is awkward, and a constant-then-decay approach lets you decide the end later.
The interactions that catch people
Batch size. Larger batches produce less noisy gradients, which tolerate and generally require a larger learning rate. Changing batch size without adjusting the peak rate is a common cause of a previously-working recipe failing.
Gradient accumulation. The schedule advances per optimiser step, not per micro-batch. Doubling the accumulation factor halves the number of optimiser steps in a run, so the schedule reaches its end in half the training — a subtle mismatch that produces a run decaying to zero far too early. See gradient accumulation.
Fine-tuning. Rates for fine-tuning are typically orders of magnitude smaller than for pretraining. Starting from a good model and applying a pretraining rate destroys what you started with — a form of catastrophic forgetting that looks like the model getting worse rather than better.
Restarts. Resuming a run must restore the schedule position along with the weights. Restarting at the peak rate after a checkpoint resume produces a visible loss spike and undoes progress.
Common mistakes
- No warmup. Early divergence, particularly with adaptive optimisers.
- Changing batch size without changing the peak rate. Recipe stops working.
- Ignoring the accumulation interaction. Schedule ends far too early.
- Pretraining rates for fine-tuning. Destroys the starting model.
- Not restoring schedule state on resume. Loss spike, lost progress.
- Tuning the shape before the peak. The peak is where most failures live.
- Decay to exactly zero too early. No learning for the tail of the run.
FAQ
How do I find the right peak rate?
A short range test — ramp the rate up over a few hundred steps and watch where loss starts rising — gives a useful upper bound quickly. Set the peak somewhat below that.
How long should warmup be?
A small fraction of total steps is the usual practice. Longer warmup for larger batches and larger models; the precise number matters less than having some.
Does the schedule matter more than the peak?
No — the peak matters more. A reasonable shape with a good peak beats a clever shape with a bad one, which is why tuning order should be peak first.
What about per-layer rates?
Useful in specific settings, particularly fine-tuning where lower layers may warrant smaller updates. Adds tuning burden, and it is not where most runs are lost.
Where to go next
For the interaction that most often breaks schedules, read gradient accumulation. For the precision issues that produce similar symptoms, mixed precision training, and for the fine-tuning context, instruction tuning.