Example-led calibration
- Industry term
Example-led calibration means showing an AI chosen examples, and sometimes counterexamples, so it can match the difficulty, tone, and format you want and tell what you would reject.
Words like “challenging” or “friendly” cover a wide range, and the model fills them in with its own average guess. An example marks where in that range you want to be. The model can't tell which shared features you chose on purpose, so it tends to copy them all.
Where you’ll meet it
You sell handmade goods online and want your assistant to write product descriptions. You paste in three you like, all for candles and all about 50 words. The new descriptions for scarves come back at 50 words each, and two of them mention a cozy glow.
What to do
Swap in samples that differ in product and length, and say what each one shows, such as the opening detail or the care line. Add one description you dislike and say why it misses. Then try the prompt on a product none of the samples covered.
Easy to mix up
| Example-led calibration | Fine-tuning |
|---|---|
| Into the prompt or saved instructions, leaving the model as it was | Into training, which changes the model itself |
Check yourself
Your prompt for meeting summaries includes five samples, and each one opens with a list of who attended. You'd rather the summaries open with what was decided. Which change gets you there more reliably?
Go deeperThe order and balance of the examples can swing the results
The technique is usually called “few-shot prompting,” after the 2020 GPT-3 paper “Language Models are Few-Shot Learners,” which showed a large model picking up a task from a handful of examples in its prompt.
A 2021 study, “Calibrate Before Use,” found that the order of examples, the balance of answers among them, and their format all changed accuracy, and that models leaned toward answers the latest examples used. Examples also travel with every request, so pasted customer emails send their private details along.