Regression evaluation
- Checking AI's work
- Standard term
A regression evaluation tries earlier requests again after an AI system changes, checking whether it can still do something it previously handled. Here, “regression” means losing an ability the system previously had.
A change can affect more than you intended. Save earlier requests and describe what each answer must get right. Include requests that once exposed problems you've fixed. Trying the old and new versions on these requests helps you compare them. Important cases may need repeated runs because AI answers vary.
Where you’ll meet it
Your team's quote-drafting tool used to include a tax line correctly. After a prompt edit to remove jargon, its quotes read more clearly. You later notice that some quotes now omit the tax line.
What to do
Save the affected quote requests with a check for the required tax line. Run those alongside requests the previous version handled correctly, and compare both versions. Keep these requests in the set you rerun after later changes.
Easy to mix up
| Regression evaluation | Eval (evaluation) |
|---|---|
| Did something that worked before stop working? | How well does this system do the task being tested? |
Check yourself
For a saved cancellation request, the old prompt succeeds 9 times in 10; the revised prompt succeeds 6 times in 10. Both pass one last run. Which reading better accounts for all these results?
Go deeperTests of new abilities can later check that those abilities still work
Anthropic's January 2026 article, “Demystifying evals for AI agents,” describes capability tests, which measure what an agent can do, becoming regression tests once pass rates are high. Tasks that once measured progress then check that later versions preserve it.
The same article calls each attempt at a task a “trial.” Comparing repeated trials before and after a change helps distinguish a new failure from one that already happened occasionally. A single passing attempt can't show you that difference.
Sources
- Demystifying evals for AI agents, Anthropic Engineering
- Evaluation best practices, OpenAI API docs