Take several models, average their weights element by element, and you would expect nonsense. Neural network weights are not obviously the sort of thing that averages meaningfully — two networks can compute the same function with entirely different internal arrangements.
Under one specific condition, averaging works: the models must be fine-tuned from the same base checkpoint. Then their weights stay close enough that the average is a valid model, frequently better than any individual member.
What changed in 2026
- Souping became a routine final step. Averaging fine-tuning runs that differ only in hyperparameters became a standard cheap improvement.
- Selection displaced naive averaging. Greedily choosing which models to include, based on held-out performance, outperformed averaging everything.
- Adapter merging borrowed the ideas. The same techniques applied to combining low-rank adapters — see adapter merging.
- Limitations stayed clear. Nobody found a way to average independently-trained models usefully.
Why the shared base matters
Fine-tuning from a common checkpoint moves weights a relatively short distance. The resulting models remain in the same broad region of the loss landscape — a connected area where the function is well-behaved, and points between two good solutions tend also to be good.
Averaging finds a point between them. Because they are in the same region, that point is a valid model, and it frequently generalises better than either — averaging cancels some of the run-specific noise each picked up.
Independently trained models have no such relationship. They may compute similar functions through completely different internal arrangements, and averaging two unrelated arrangements produces something that computes neither. The result is not a slightly worse model; it is not a model.
The practical rule: same base checkpoint, or do not average.
| Scenario |
Averaging works? |
| Same base, different hyperparameters |
Yes |
| Same base, different data ordering |
Yes |
| Same base, different fine-tuning data |
Often, with interference |
| Different base checkpoints |
No |
| Independently pretrained |
No |
Greedy selection
Averaging every candidate you have is the simple approach and rarely optimal, because a poor run drags the average down.
The better procedure is greedy: start with the best individual model, then consider adding each remaining candidate one at a time, keeping it only if the average improves held-out performance. Continue until nothing improves it.
That reliably outperforms averaging everything, and it needs a held-out evaluation set you have not tuned against — the same requirement as any selection procedure, per golden datasets.
Weighted averaging, where better models contribute more, is a further refinement with diminishing returns relative to the added complexity.
Free compared with an ensemble
The comparison that makes souping attractive.
An ensemble runs every member model on every request and combines their outputs. It works well and costs proportionally — five models means five times the inference.
A soup is one model. Inference cost is identical to a single model, because it is a single model. You captured some of the ensemble benefit at no serving cost.
The gain is smaller than a true ensemble's, and it is free. For anything cost-sensitive — which is most production serving — a modest free improvement beats a larger improvement at five times the cost.
The other practical use is combining fine-tuning runs that differ only in seed or minor hyperparameters. Rather than picking the best and discarding the rest, average the good ones and get something slightly better than any.
Common mistakes
- Averaging across different base checkpoints. Produces a non-model.
- Averaging everything. A poor run degrades the result.
- Selecting on the training set. Overfits the selection.
- Expecting ensemble-scale gains. Souping captures part of the benefit.
- Merging adapters trained for conflicting objectives. Interference — see adapter merging.
- Not evaluating each capability separately. Interference is quiet in aggregate scores.
FAQ
How much improvement should I expect?
Modest — a small gain rather than a transformation. It is worth doing because it is free at inference, not because it is large.
Can I soup models fine-tuned on different tasks?
Sometimes, with interference. You may get partial capability from each rather than full capability from both, and it needs per-capability evaluation to detect.
Does this work for adapters?
Yes, and the same interference caveats apply — see adapter merging.
Is it the same as checkpoint averaging?
Closely related. Averaging checkpoints from different points in one training run is a special case with the same underlying reasoning, and it is a common regularisation technique.
Where to go next
For merging adapters specifically, read adapter merging. For the fine-tuning runs souping combines, QLoRA, and for the held-out evaluation selection requires, golden datasets.