Instruction tuning
- Training
- Standard term
Instruction tuning is fine-tuning a model on examples of requests paired with good responses, so it more often treats what you type as a task to carry out.
Pretraining, the first and largest stage of training, teaches a model to continue text, so a bare request can look like the first line of a list. Fine-tuning trains it further on far fewer examples, written by people or models or pulled from existing text. The model still predicts the next word, but after a request the likeliest words form an answer.
Where you’ll meet it
You want a free model on your laptop so your notes never leave it. The download page lists a base version and an instruct version. You try the base one, ask it to summarize a page of notes, and it writes more lines that read like the notes.
What to do
Switch to the instruct version. Then try it on a few pages you know well and read each summary against them, since the label says how the model was trained and nothing about how well it handles your notes.
Easy to mix up
Pretraining comes first, training a model from scratch on huge amounts of text. Instruction tuning comes afterward, teaching that model to treat what you type as a task to carry out.
Check yourself
A company tunes a freely downloadable model on a thousand example requests and answers. Some requests are in English and some in Spanish, and every answer is in Spanish. What will the tuned model most likely do with a new English request?
Go deeperWhich examples are chosen changes what tuning teaches
Google's FLAN research found that a model tuned on many kinds of written instructions did better on kinds of task it was never tuned on. OpenAI's InstructGPT made the stage widely known by pairing it with preference training, a further stage that steers the model toward answers people ranked higher. People in that study preferred a model put through both stages over an untuned one more than a hundred times larger.
The quality of the examples can count for more than their number. Meta's LIMA study tuned a large model on about a thousand carefully chosen examples and reported answers competitive with assistants tuned on far more data.
Examples that go beyond what the base model, the version before tuning, already knows can backfire. One study found that tuning on answers to facts the base model never learned made it more likely to invent answers, as if it had learned to sound sure past the edge of its knowledge.