Held-out examples
- Checking AI's work
- Standard term
Held-out examples are cases you set aside while choosing or improving an AI system or its instructions. You test with them later to see how the finished version handles cases that didn't guide your choices.
If you repeatedly revise a prompt to pass the same cases, it may handle those cases well while struggling with others. For a final test, use cases whose results haven't influenced your revisions or your choice of version. Once you use a result to make either decision, save new cases for the next final test.
Where you’ll meet it
You've spent an afternoon improving a prompt that converts call notes into customer records. It now handles your five sample calls correctly, including their awkward details. You're about to give it next week's calls.
What to do
Try the calls you set aside before revising the prompt. If their results lead you to change it, add those calls to the examples you use when making revisions. Use a different group, whose results you haven't seen, for the next final test.
Easy to mix up
| Held-out examples | Example-led calibration |
|---|---|
| Cases reserved for testing, with answers used to check the results | Examples included in the request to show the model what you want |
Check yourself
A team tries four finished prompts on 30 emails kept aside. It launches the prompt with the highest score. A manager wants to check how this winner will handle tomorrow's emails. Which score gives a better check?
Go deeperMachine learning distinguishes development checks from a final test
Google's Machine Learning Crash Course distinguishes training data, examples used to train a model; validation data, examples checked while choosing improvements; and a test set saved for the end. A final test is less independent if its results helped you choose the system, even when those cases weren't part of training.
Sources
- Datasets: Dividing the original dataset, Google Machine Learning Crash Course
- Machine Learning Glossary, Google for Developers