Parameter-efficient fine-tuning, or PEFT, is a family of techniques that adapt a pretrained model to a new task by training a small number of additional parameters while leaving the vast majority of the original weights frozen. LoRA is the best-known method in this family, but it is one option among several, including prefix-tuning, prompt-tuning, and adapter layers. The practical result is similar across all of them: most of the behavior change of full fine-tuning at a small fraction of the memory, storage, and compute cost, in exchange for a ceiling on how much the model can actually be made to change.
How it works
Full fine-tuning updates every weight in the model, which for a modern large language model means tracking and storing gradients for tens or hundreds of billions of parameters. That requires enormous GPU memory, produces a full new copy of the model per task, and makes experimentation slow and expensive.
PEFT methods instead insert a small number of new, trainable parameters into or alongside the frozen network. LoRA, for example, adds a pair of small low-rank matrices next to selected weight matrices, typically inside the attention layers, and only trains those matrices. The rank of those matrices, often written as r, controls how many new parameters get added. The original weights never move at all; the small matrices simply supply an additive correction on top. Because the frozen weights do not need gradients tracked, memory use drops sharply, and because the trained artifact is just the small matrices, an organization can store dozens of task-specific adapters at a few megabytes each rather than dozens of full model copies at hundreds of gigabytes each.
PEFT methods compared
| Method |
What gets added |
Trainable parameters vs full model |
Typical use case |
| LoRA |
Low-rank matrices next to weight layers |
Roughly 0.1-1 percent |
General-purpose task and style adaptation |
| QLoRA |
LoRA on top of a quantized base model |
Similar to LoRA, lower memory footprint |
Fine-tuning large models on one consumer GPU |
| Prefix-tuning |
Trainable tokens prepended to each layer input |
Under 0.1 percent |
Lightweight task switching, multi-task serving |
| Prompt-tuning |
Trainable embedding vectors prepended to the input |
Well under 0.1 percent |
Simple task steering with minimal storage |
| Adapter layers |
Small bottleneck layers inserted between existing layers |
Roughly 1-4 percent |
Modular per-task or per-domain customization |
QLoRA deserves a specific mention because it stacks two efficiency techniques at once: the frozen base model is quantized down to 4-bit precision, and LoRA adapters are trained on top in higher precision. That combination is what made fine-tuning a 30 to 70 billion parameter model on a single high-end consumer GPU realistic rather than theoretical. For more on the quantization half of that pairing, see our explainer on quantized models.
What PEFT actually changes versus full fine-tuning
The honest tradeoff is capacity versus cost. Full fine-tuning can reshape model behavior more thoroughly because every weight is available to move, which matters for tasks far from the pretraining distribution, such as teaching an entirely new language, an unusual output format, or specialized domain knowledge that conflicts with what the model already encodes. PEFT methods, by only adjusting a thin slice of the network, tend to excel at steering tone, style, and narrow task behavior, and at teaching a model to follow a specific format reliably, but they generally struggle to inject large amounts of genuinely new knowledge as cleanly as full fine-tuning or continued pretraining can.
In most real deployments this is not actually a close call: PEFT is the default choice unless there is a specific reason to need full fine-tuning, because the cost difference is not marginal. It is often the difference between needing one GPU for a few hours versus a cluster for days.
Common mistakes
- Choosing a rank that is too low for the task. A very small LoRA rank saves memory but can underfit tasks that need real behavioral change; most teams need to test a couple of rank values rather than assuming the smallest one works.
- Expecting PEFT to teach genuinely new facts reliably. PEFT is strong at style, tone, and format; injecting large amounts of new factual knowledge tends to work better through retrieval or full fine-tuning instead.
- Stacking multiple adapters without testing interactions. Several LoRA adapters applied to the same base model can interact in unexpected ways; merged behavior should always be tested, not assumed.
- Skipping a before-and-after comparison against the frozen base model. Without that baseline, it is easy to ship an adapter that improves style while quietly degrading accuracy on edge cases.
FAQ
Is LoRA the same thing as PEFT?
No. LoRA is one specific PEFT method. PEFT is the umbrella term for any technique that adapts a model by training a small number of new parameters while freezing most of the original weights.
Does PEFT produce a worse model than full fine-tuning?
Not necessarily worse, but different. For most style, tone, and narrow-task adaptation, well-tuned PEFT methods get close to full fine-tuning quality. For deep, broad knowledge changes, full fine-tuning still tends to have an edge.
Can PEFT be combined with quantization?
Yes, this is exactly what QLoRA does, and it is one of the main reasons fine-tuning large open models on modest hardware became practical for individual developers and small teams.
Do PEFT adapters work across different base model versions?
Generally no. An adapter is trained against the specific weight structure of one base model, so upgrading the base model typically means retraining the adapter rather than simply reattaching it.
Where to go next
For related efficiency and deployment tradeoffs, see our guides to quantized models, running quantized AI models, and open-source vs closed-source LLMs.