Distillation is how a lot of today's small, capable models are made. The idea is simple: instead of training a small model from scratch on raw data alone, you train it to imitate a larger, stronger model, the teacher. The small student model inherits much of the teacher's behaviour while being far cheaper and faster to run.
For builders, distillation is also a practical technique: if a large model handles your task well but costs too much at your volume, you can often distil that behaviour into a small model you control. This guide explains how distillation works, the main variants, when it beats prompting or ordinary fine-tuning, and what to check before you start.
The original idea: learning from soft labels
The modern concept comes from Hinton, Vinyals and Dean (2015), "Distilling the Knowledge in a Neural Network." Their insight was that a trained model's full output distribution contains more information than the single correct label.
Suppose an image classifier sees a picture of a husky. The label says "husky." But the teacher's probabilities might say 80% husky, 15% wolf, 4% malamute and almost 0% car. Those soft labels encode similarity: a husky is like a wolf and nothing like a car. Training a student on these probabilities, often softened further with a temperature setting, teaches it the teacher's sense of how classes relate, which helps it generalise from less data.
For language models, the equivalent is training the student to match the teacher's probability distribution over the next token, not just the single token that appeared in the text.
Three ways to distil a language model
| Approach | What the student learns from | Needs teacher internals? | Typical use |
|---|---|---|---|
| Logit (soft-label) distillation | Teacher's full next-token probabilities | Yes (probabilities) | Labs training model families |
| Sequence-level distillation | Complete outputs generated by the teacher | No, only outputs | Most practical for builders |
| Feature distillation | Teacher's internal representations | Yes (hidden states) | Research, architecture-matched pairs |
Logit distillation
The student is trained to match the teacher's probability distribution at each position. This transfers the most information per example, but requires access to the teacher's probabilities, so it is mainly used by labs distilling their own models, for example when producing smaller models in a family from a larger one.
Sequence-level distillation (synthetic data)
The teacher generates complete answers to a large set of prompts, and the student is fine-tuned on those prompt-answer pairs. This only needs the teacher's text outputs, which makes it the approach most teams use. Kim and Rush (2016), "Sequence-Level Knowledge Distillation," showed this works well for translation, and it has since become a standard way to produce instruction-following models.
Feature distillation
The student also learns to match internal activations of the teacher. It can transfer more, but requires compatible architectures and access to internals, so it is less common in practice.
When distillation makes sense for your product
Distillation is worth considering when all of these are true:
- A large model already does the task well, and you have measured it.
- Volume is high, so per-request savings add up.
- The task is narrow and stable: classification, extraction, a specific writing style, a domain Q&A format. Distilling general intelligence into a small model does not work; distilling a well-defined behaviour does.
- Latency, cost or data control matter enough to justify maintaining your own model.
If the task is narrow but you only need modest gains, try prompting a small model with good examples first; see small models, big jobs.
A practical distillation workflow
- Define the task and the quality bar. Build an eval set with 200 or more examples, scored the way you will score in production.
- Collect diverse inputs. Use real inputs if you are permitted to; otherwise generate realistic ones. Diversity matters more than volume. Cover edge cases deliberately.
- Generate teacher outputs with your best prompt, at low temperature for consistency.
- Filter. Validate every teacher output: schema checks, rule checks, a grader model or human spot checks. Training on the teacher's mistakes teaches the student to repeat them.
- Fine-tune the student, typically with parameter-efficient methods such as LoRA. A few thousand high-quality examples often suffice for narrow tasks.
- Evaluate against the teacher on the held-out eval set, per category, and read the disagreements.
- Deploy with a fallback. Route low-confidence or failed-validation cases to the teacher.
- Refresh periodically as inputs drift, using production samples (with permission) and new teacher outputs.
After distillation, you can quantize the student for even cheaper serving, re-checking quality afterwards.
What students lose
A student rarely matches the teacher everywhere. Expect:
- weaker performance outside the training distribution: unusual inputs, new topics, other languages;
- less robust reasoning on harder instances of the task;
- inherited teacher errors and biases, unless filtered out;
- a narrower personality: students trained on one style can become rigid.
These are acceptable trade-offs for narrow, high-volume tasks with a fallback, and unacceptable for open-ended assistants without one.
How to tell whether distillation worked
Distillation projects fail quietly when the evaluation is too easy. A student that matches the teacher on the same kind of inputs it was trained on can still collapse on the inputs real users send. Four checks catch most problems:
- Held-out inputs from a different time period. If your training inputs came from last quarter, test on this month's traffic. Drift in topics and phrasing shows up immediately.
- A hard slice. Keep a set of difficult and unusual inputs that were never used for training. The gap between student and teacher is usually largest here, and it tells you how much traffic to keep on the fallback path.
- Agreement rate with the teacher, measured per category. Overall agreement can look high while one important category is badly wrong.
- Calibration of the fallback signal. Check that the cases the student flags as low-confidence really are the ones it gets wrong. A fallback that triggers on the wrong cases gives you the cost of both models and the quality of neither.
Check the terms before you distil
Many commercial model providers' terms of service restrict using their outputs to train models that compete with them, and terms differ between providers and change over time. Before distilling from a hosted model, read the current terms for the specific model and service tier, and get legal advice if in doubt. Open-weights models come with their own licences, some of which also restrict how outputs or derivatives may be used. Also check that your input data can be used for training under your privacy policy and agreements with customers.
Distillation vs. fine-tuning vs. preference tuning
These terms overlap. Fine-tuning is the general process of continuing to train a model on new data. Distillation is fine-tuning where the data, or the target probabilities, come from a teacher model. Preference tuning (RLHF, DPO) trains a model on comparisons between better and worse responses; see preference tuning: RLHF and DPO side by side. Many production pipelines use all three: distil for capability, then preference-tune for style and safety.
Frequently asked questions
What is knowledge distillation in machine learning?
Knowledge distillation trains a smaller student model to reproduce the behaviour of a larger teacher model, either by matching the teacher's output probabilities or by fine-tuning on outputs the teacher generated.
How much data do I need to distil a model for one task?
For narrow tasks, a few thousand high-quality, diverse, validated examples are often enough to get close to the teacher. Broader tasks need far more, and general-purpose capability cannot be fully distilled into a small model.
Is it legal to train a model on another model's outputs?
It depends on the provider's terms of service or the model's licence, which often restrict using outputs to build competing models. Check the current terms for the specific model and get legal advice where needed.
Is distillation better than fine-tuning on human-labelled data?
They complement each other. Human labels are the gold standard but costly; teacher-generated data is cheap and plentiful. A common approach is to distil with teacher outputs and validate or correct a sample with humans.
