You have a task a large model handles well and a volume that makes calling it expensive. Distillation is the response: use the large model to generate training data, fine-tune a much smaller model on it, and deploy the small one. Inference cost drops substantially, latency improves, and the small model does one thing nearly as well as the large one did.
The constraint is in that last phrase. One thing.
What changed in 2026
- Licensing became the first check. Terms prohibiting using outputs to train competing models became widely enforced and widely noticed, moving licence review to the start of the process.
- Small models improved independently. Better small models raised the baseline, which meant distillation had to beat a stronger starting point to be worthwhile.
- Data coverage got emphasized. Recognition spread that distillation quality is determined mainly by whether the generated dataset covers the real input distribution.
- Task narrowing became the advice. Attempts at general-purpose distillation consistently disappointed, pushing practice toward well-scoped tasks.
Where distillation works
| Task type |
Distils well |
| Classification into fixed categories |
Very well |
| Structured extraction from documents |
Well |
| Routing and intent detection |
Very well |
| Domain-specific rewriting |
Well |
| Short-form summarization |
Reasonably |
| Simple tool selection |
Reasonably |
| Multi-step reasoning |
Poorly |
| Open-ended assistance |
Poorly |
| Tasks requiring broad world knowledge |
Poorly |
The pattern is that narrow, well-defined tasks with a bounded output space distil cleanly, and open-ended capability does not. A small model can learn to classify support tickets nearly as well as a large one. It cannot learn to be a general assistant from a large one's outputs, because the capability being transferred is not a narrow function.
Building the pipeline
Check the licence first. Many commercial model terms prohibit using outputs to train models, particularly competing ones. Open-weight models vary — the analysis in open-weight model licensing applies. This is a five-minute check that occasionally ends the project, and it should happen before the work rather than after.
Establish the baseline. Measure a small model with a good prompt on your task before distilling anything. Small models have improved enough that they sometimes already meet the bar, in which case distillation is unnecessary work.
Generate data covering the real distribution. This is where quality is determined. Sample inputs from actual production traffic rather than inventing them, and make sure edge cases and rare categories are represented — a distilled model is very good on the distribution it saw and poor outside it.
Include the reasoning where it helps. For tasks with intermediate steps, training on the teacher's reasoning as well as its final answer transfers more than the answer alone.
Filter the teacher's output. The teacher is not perfect, and training on its errors teaches them. Where verification is possible — tests, schema checks, a second model's agreement — filter before training.
Evaluate against the teacher, not against perfection. The relevant question is how much capability was retained and whether the gap is acceptable at the cost saving achieved.
Common mistakes
- Skipping the licence check. Occasionally fatal, always cheap to do first.
- No small-model baseline. May already be sufficient.
- Training data that does not match production inputs. Good on the distribution it saw, poor outside it.
- Not filtering teacher errors. Teaches the mistakes along with the capability.
- Distilling an open-ended task. Consistently disappointing.
- Evaluating only on average performance. Check the rare categories; that is where distilled models fall down.
FAQ
How much smaller can the student be?
A large ratio is achievable on narrow tasks. The right size is whatever meets your quality bar, found by trying a couple of options.
Do I need the teacher's internal probabilities?
Classical distillation uses them and API access rarely provides them. Training on generated outputs alone works well for the task-specific case.
How much data do I need?
More than for adaptation fine-tuning, since you are transferring capability rather than adjusting style. Coverage of the input distribution matters more than raw count.
Is this the same as fine-tuning?
Distillation is fine-tuning where the training data came from another model. The mechanics in LoRA fine-tuning guide apply to the training step.
Where to go next
For the underlying technique, read model distillation explained and LoRA fine-tuning guide. For licence constraints, open-weight model licensing.