You have three fine-tunes of the same base model, each good at a different thing, and you want one model that does all three. The obvious approach is retraining on the combined data, which costs time, compute, and requires having all the data. Merging skips that entirely: it combines the weights directly, arithmetically, in minutes.
That it works at all is somewhat surprising. That it works reasonably well is what made it a standard technique.
What changed in 2026
- Merging became routine in open-model releases. A large share of community models on public hubs were produced by merging rather than by training, which normalized the technique.
- Task-vector methods displaced averaging. Treating a fine-tune as a direction from the base, then combining directions with interference resolution, clearly outperformed naive weight averaging.
- Tooling matured. Merge configurations became declarative and reproducible rather than ad hoc scripts.
- Evaluation caught up. Recognition that merged models need evaluating on every constituent task, not just the headline one, became standard advice.
The methods
| Method |
How it works |
Best for |
| Simple averaging |
Mean of corresponding weights |
Models fine-tuned from the same base on similar tasks |
| Weighted averaging |
Mean with per-model weights |
When one task matters more |
| Task arithmetic |
Add and subtract fine-tune directions from the base |
Combining or removing capabilities |
| Interference-resolving merges |
Trim small changes, resolve sign conflicts, then combine |
Multiple diverse fine-tunes |
| Layer-wise merging |
Different sources for different layers |
Experimental; occasionally strong |
Task arithmetic is the conceptually interesting one. The difference between a fine-tuned model and its base is a vector in weight space representing what the fine-tune taught. Those vectors can be added to combine capabilities, scaled to strengthen or weaken them, and in principle subtracted to remove a behaviour.
Interference resolution addresses the main problem with naive combination: when two fine-tunes moved the same weight in opposite directions, averaging cancels both to something that helps neither. Trimming negligible changes and resolving conflicting directions before combining preserves more of each contribution.
When to merge and when not to
Merge when you want one artifact instead of several and can accept a small quality cost per task. One model is simpler to deploy, cheaper to serve, and needs one set of monitoring.
Do not merge when you can serve adapters separately. If your fine-tunes are LoRA adapters over a shared base, a serving stack can hold many and select per request, preserving each task's quality entirely. That is strictly better when it is available, and it is the pattern described in PEFT methods explained.
Do not merge across different base models. The technique depends on weights being in a comparable space, which requires a shared ancestor.
And evaluate on everything. The characteristic merge failure is a model that is excellent at the task you checked and quietly worse at the two you did not. Run the full evaluation set for each constituent capability before shipping — the discipline in eval-driven development for AI.
Common mistakes
- Merging models with different bases. Does not work; the weights are not comparable.
- Evaluating only the primary task. Regressions hide in the others.
- Naive averaging for diverse fine-tunes. Interference cancels contributions.
- Merging when adapter swapping is available. Gives up quality for no benefit.
- Treating merge weights as arbitrary. They matter and are worth a small search.
- Assuming merged capabilities compose cleanly. Sometimes they interact badly in ways only evaluation reveals.
FAQ
Why does merging work at all?
Fine-tuning from a shared base tends to make relatively small, structured adjustments, so the resulting weights remain in a region where interpolation is meaningful. It is an empirical finding more than a derived one.
Can I merge models of different sizes?
No. Merging requires matching architectures and parameter shapes.
How many models can I merge?
Several works; quality degrades as you add more, particularly with naive methods. Interference-resolving approaches hold up better.
Is merging cheaper than training?
Dramatically. It is weight arithmetic, requiring no training data and minutes of compute rather than hours on accelerators.
Where to go next
For producing the fine-tunes, read LoRA fine-tuning guide and PEFT methods explained. For evaluating the result, eval-driven development for AI.