Full fine-tuning updates every parameter, which for a large model means holding the weights, the gradients, and the optimizer state simultaneously — several times the model size in memory. Parameter-efficient fine-tuning is the collective name for techniques that avoid this by freezing the bulk of the model and training something small alongside it.
They all share that shape. Where they differ is what the small trained thing is.
What changed in 2026
- LoRA consolidated as the default. Ecosystem support, tooling maturity, and predictable results made it the method most teams reach for without deliberation.
- Quantized training became routine. Loading the frozen base at reduced precision while training full-precision adapters brought large models within single-accelerator reach.
- Variants refined rather than replaced. Newer methods improved on LoRA at the margins rather than displacing it, which is why the default stayed stable.
- Serving many adapters became normal. Infrastructure for hot-swapping adapters over a shared base model matured into standard deployment tooling.
The main approaches
| Method |
What it trains |
Memory |
Quality |
Notes |
| Full fine-tuning |
Every parameter |
Very high |
Best possible |
Rarely justified for adaptation |
| LoRA |
Low-rank matrices beside attention weights |
Low |
Very good |
The sensible default |
| Quantized LoRA |
Same, over a quantized frozen base |
Lowest practical |
Good |
Enables consumer hardware |
| Weight-decomposed variants |
Magnitude and direction separately |
Low |
Slightly better than LoRA |
Marginal gain, less tooling |
| Adapter layers |
Small inserted modules between layers |
Low |
Good |
Adds inference latency |
| Prompt and prefix tuning |
Learned tokens prepended to input |
Lowest |
Weakest |
Very cheap; many variants per base |
The practical recommendation is unglamorous: use LoRA. It is well supported, behaves predictably, and its adapters can be merged back into base weights so inference carries no extra latency. If memory is the constraint, use the quantized variant. Everything else is an optimization to consider after you have a working result.
The adapter-layer row deserves a caveat. Inserted modules add computation at inference time, unlike LoRA adapters which can be folded into the base weights. For a latency-sensitive service that difference matters.
Choosing by constraint
If your constraint is memory, quantized LoRA is the answer and the quality cost is modest for most adaptation tasks.
If your constraint is inference latency, use LoRA and merge the adapter into the base weights before deploying. A merged model has identical inference cost to the original.
If your constraint is serving many task variants, keep adapters unmerged and swap them per request. Adapters are small, and a serving stack can hold many over one loaded base.
If your constraint is quality and you have real budget, full fine-tuning still wins on the hardest adaptation tasks — and the gap has narrowed enough that it is rarely worth the cost.
Whatever you pick, the data matters more than the method. The guidance in LoRA fine-tuning guide on curating a small high-quality dataset applies to every method here, and switching methods will not rescue bad training data.
Common mistakes
- Method shopping before a baseline. Get LoRA working, then optimize.
- Expecting a method change to fix data problems. It will not.
- Using inserted adapters in a latency-critical path. Merge-able methods avoid the overhead.
- Prompt tuning for hard behavior changes. It has the least capacity of the family.
- Ignoring base model lifecycle. Every adapter is bound to the base it was trained on.
FAQ
Is quantized training much worse?
For typical adaptation tasks the difference is small. It is more noticeable on tasks requiring precise numerical behavior.
Can I merge a LoRA adapter permanently?
Yes, and you should for single-task deployment. Merging removes inference overhead entirely.
How many adapters can one server hold?
Adapters are small relative to base weights, so a serving stack can typically hold many. The practical limit is usually the serving framework rather than memory.
Does PEFT work for very small models?
It works, and the benefit shrinks — full fine-tuning a small model is already cheap, so the memory saving matters less.
Where to go next
For the default method in depth, read LoRA fine-tuning guide. For combining trained models, model merging explained, and for whether to fine-tune at all, RAG vs fine-tuning.