Training / Standard term
Distillation
Training a student model to match outputs, probabilities, representations, or behaviors supplied by a teacher model or ensemble.
Distillation can use generated examples, soft probability targets, intermediate activations, or combinations of these. The student may be smaller, faster, or specialized, but quality and savings depend on architecture, data, task, and deployment hardware. It can match the teacher on a target distribution while differing sharply elsewhere.
Builder example
A distilled model is one possible explanation for strong small-model performance, not the default explanation. Deployment routing should compare models directly on ordinary and difficult cases, including failures that the teacher may have generated into the training data.
A frontier model labels thousands of support tickets, then a smaller model learns to reproduce the labels cheaply.
Audit the labels, sample mistakes, and keep the stronger model for ambiguous cases.
Common confusion: A student can inherit teacher errors, correct some through other data, or introduce new ones. Distillation does not create a guaranteed subset of teacher behavior.