Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Distillation

  • Training
  • Standard term
2 steps

Distillation trains a smaller or faster model, called the “student,” to reproduce the outputs or behavior of a larger model, called the “teacher,” so it can do a similar job at lower cost.

In the plainest version, the teacher answers many questions and the student trains on its answers. A richer version also uses how sure the teacher was of each possible answer. Those odds show which wrong answers came close, so the student learns more from each example than a bare right-or-wrong label would teach it.

Where you’ll meet it

An AI company announces a “mini” version of its main model. The post says it was trained partly on the larger model's answers, matches it on everyday tasks, and costs a tenth as much to use.

What to do

Read “matches” as “matches on the tests the company ran.” Before moving your own work to the mini model, run a handful of your real tasks through both, including the odd ones you rarely see, and compare the answers side by side.

Easy to mix up

Quantization keeps the same model and stores its numbers less precisely, so it runs cheaper. Distillation trains a new student model, which can differ from the teacher in size and design.

Check yourself

Your team distilled a small model from a large one, using the large model's answers to thousands of customer questions. The large model often gets one kind of refund question wrong, and so does the student. Which fix is more likely to work?

Go deeperDistillation began as model compression and now passes on reasoning too

Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil showed in 2006 that a small model could be trained to copy a large group of models, and called it model compression. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean named it distillation in 2015 and showed it improving a widely used commercial speech recognition system.

Many recent students also train on the teacher's written step-by-step reasoning, which can carry a good share of its problem-solving into far smaller models. A small model's strong results don't prove it was distilled, since better data and longer training also help. Several AI providers' terms of use forbid using their outputs to train a competing model.

Practice Modern AI

Or take the learning quiz on a mix of every set