Fine-tuning a small language model in 2026 is an afternoon project, not a research endeavour. With LoRA adapters, QLoRA quantisation, and mature training libraries, a 7B model can be specialised on a consumer-grade GPU in 2–4 hours for under $20 in cloud compute. Here is the end-to-end workflow.
What changed in 2026
- Base models improved dramatically. Llama 4 Scout (3B) and Phi-4 Mini (3.8B) punch well above their size, meaning you often need fewer fine-tuning examples to reach production quality.
- LoRA became the default fine-tuning method. Full fine-tuning a 7B model requires 80+ GB VRAM; LoRA + 4-bit quantisation fits the same model into 12 GB.
- Unsloth and TRL dramatically cut training time. Both libraries are now the go-to for efficient fine-tuning.
- Model merging is practical. You can merge LoRA adapters from different task specialists into one model without retraining.
Decide if fine-tuning is the right tool
Fine-tuning changes how the model behaves (format, style, domain vocabulary). It does not add new knowledge efficiently. If your problem is "the model gives wrong factual answers," fine-tuning will not fix it — RAG will.
| Goal |
Better approach |
| Model always outputs JSON schema |
Fine-tune (format) |
| Model knows your domain vocabulary |
Fine-tune (style) |
| Model answers from your docs correctly |
RAG |
| Model follows complex multi-step instructions |
Prompt engineering first |
| Model classifies into your custom categories |
Fine-tune (classification) |
Step 1: Prepare your data
LoRA fine-tuning needs instruction-response pairs in JSONL format:
{"prompt": "Classify this support ticket: 'My invoice is wrong'", "completion": "billing"}
{"prompt": "Classify this support ticket: 'App crashes on login'", "completion": "bug"}
Rules for good training data:
- Minimum ~200 examples for format tasks; ~1,000+ for domain adaptation.
- Consistent output format — every example should show exactly what you want.
- 20% held-out validation set — never train and evaluate on the same data.
- Deduplication — near-duplicates inflate training curves without improving generalisation.
Step 2: Fine-tune with Unsloth + TRL
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-4-scout-3b-instruct",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16, # LoRA rank — higher = more capacity, more VRAM
lora_alpha=16,
target_modules=["q_proj", "v_proj"],
lora_dropout=0,
)
dataset = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
args=TrainingArguments(
output_dir="./output",
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=2e-4,
fp16=True,
),
)
trainer.train()
model.save_pretrained("./my-adapter")
Total training time for 1,000 examples on an A10G: ~30–90 minutes.
Step 3: Evaluate properly
Run your held-out validation set through the fine-tuned model and measure task-specific metrics — not perplexity.
correct = 0
for item in val_set:
pred = model.generate(item["prompt"])
if pred.strip() == item["expected"]:
correct += 1
accuracy = correct / len(val_set)
Compare against: (a) the base model without fine-tuning, (b) the base model with a few-shot prompt. Fine-tuning should beat both by a meaningful margin to justify the maintenance cost.
How to start
- Try prompt engineering first. If 5-shot prompting reaches 85% accuracy, fine-tuning to reach 88% may not be worth the ops cost.
- Pick a model close to your target size — Llama 4 Scout for resource-constrained, Mistral 7B for balance.
- Start with 200–500 high-quality examples before scaling.
- Validate on your task metric, not on generic benchmarks.
- Log all experiments — learning rate, rank, epochs, dataset size.
Common mistakes
Training on the full dataset without a validation split. You will overfit without knowing it.
High learning rate. Fine-tuning usually needs 1e-4 to 3e-4. Higher rates cause catastrophic forgetting.
Using LoRA rank too low. r=4 is too low for complex tasks. Start at r=16 and lower only if you are VRAM-constrained.
Evaluating with the same examples you trained on. Contamination produces misleadingly high scores.
Skipping the base model baseline. You need to know what the fine-tuned model gains relative to the base.
What to skip
- Full fine-tuning unless you have 4+ A100s and a clear need — LoRA matches or exceeds full fine-tuning on most task-specific benchmarks.
- RLHF from scratch — use DPO (Direct Preference Optimisation) if you need preference alignment; it is far simpler.
- Fine-tuning a model larger than you can serve — a 70B fine-tuned model you cannot deploy serves no one.
FAQ
How much data do I need?
For format/style tasks: 200–500 examples. For domain adaptation: 1,000–5,000. For general instruction tuning: 50k+. Quality always beats quantity.
Can I fine-tune a closed API model?
OpenAI and Anthropic both offer fine-tuning endpoints for some models. Costs are higher than self-hosted but require zero infra.
Will fine-tuning hurt general capabilities?
Yes, slightly — this is called "catastrophic forgetting." LoRA minimises it; full fine-tuning is worse. Keep the adapter separate and merge only for the specific deployment.
How long does a LoRA fine-tune take?
On an A10G (24 GB): 1,000 examples, 3 epochs, 7B model = ~45 minutes with Unsloth.
Where to go next