Model distillation is how you take the skill of a large, expensive model and pour it into a small, cheap one. A big "teacher" model generates outputs, and a smaller "student" model trains to reproduce them. Done well, the student keeps most of the teacher's quality while running far faster and cheaper. It is one of the main reasons a phone-sized model in 2026 can feel close to a model that needed a data center a few years ago.
What changed in 2026
- Distillation became the default way to ship small models. Most compact production models are now distilled from a larger sibling rather than trained from scratch.
- Synthetic teacher data got standardized. Generating large sets of teacher responses to train students is now a routine pipeline, overlapping heavily with synthetic data work.
- Combining distillation with quantization became common. Teams distill first for skill, then quantize for size, stacking the two compression techniques for the smallest usable model.
How distillation works
The key idea is that a teacher's full output distribution contains more information than a plain label. If the correct answer is "cat," a hard label says only "cat = 1, everything else = 0." The teacher instead says something like "cat 0.85, dog 0.10, fox 0.05," which quietly tells the student that dogs are closer to cats than foxes are. These softened probabilities are the "dark knowledge" that makes distillation work.
There are a few flavors:
- Response-based. The student mimics the teacher's final outputs or token probabilities. Simple and by far the most common for language models.
- Feature-based. The student matches the teacher's internal representations, not just the final answer, which can transfer more but needs access to the teacher's internals.
- Data distillation. The teacher generates a large synthetic dataset and the student trains on it as ordinary examples, which is flexible and works even through an API.
Distillation vs other compression
Distillation is often confused with quantization and pruning. They solve the same problem differently.
| Technique |
What it does |
Keeps original weights? |
Best when |
| Distillation |
Trains a new smaller model to imitate a teacher |
No, new model |
You want a much smaller architecture |
| Quantization |
Stores existing weights in lower precision |
Yes, same model |
You want smaller and faster with minimal effort |
| Pruning |
Removes unimportant weights or neurons |
Mostly, trimmed |
You have redundancy to cut |
If you have not compared the alternatives, start with quantization explained; it is usually the cheaper first move before you commit to full distillation.
When distillation is worth it
Distillation pays off when you need a permanently smaller footprint: on-device inference, high-volume low-latency serving, or strict cost ceilings. It also lets you capture a proprietary model's behavior into a model you own and can serve yourself. It is not worth it when a quantized version of the original already meets your latency and cost targets, since distillation requires a training run and careful evaluation that quantization does not.
Common pitfalls
- Expecting the student to beat the teacher. The teacher is the ceiling. A student can approach it, occasionally match it on narrow tasks, but generally will not exceed it.
- Distilling the teacher's mistakes. If the teacher hallucinates or is biased, the student learns those flaws faithfully. Filter teacher outputs first.
- Evaluating only on average accuracy. Students often lose the most on rare and hard inputs. Measure the tails, not just the mean.
- Skipping a real test set. Because the training signal comes from the teacher, it is easy to overfit to teacher quirks. Evaluate against ground truth.
FAQ
Is distillation the same as fine-tuning?
No. Fine-tuning adapts a model to a task using labeled data. Distillation trains one model to imitate another. You can combine them by fine-tuning the student on distilled data.
Can I distill a model I only access through an API?
Often yes, using data distillation: send prompts, collect the teacher's responses, and train your student on those pairs. Check the provider's terms, since many restrict using outputs to train competing models.
Does distillation reduce accuracy?
Usually a little. A well-built student keeps most of the teacher's quality, but expect some loss, concentrated on the hardest cases.
How is this related to speculative decoding?
Both use a small model alongside a large one, but the goals differ. Distillation replaces the big model; speculative decoding uses a small model to speed up the big one without replacing it.
Where to go next