Precision and recall
- Checking AI's work
- Standard term
Precision is the percentage of an AI's results that match what you're looking for. Recall is the percentage of all matching items that the AI found.
A stricter rule for selecting items can reduce false alarms while missing more items you want. A looser rule can find more wanted items while including more unwanted ones. The useful balance depends on how much each kind of mistake costs you.
Where you’ll meet it
Of the 20 invoices your AI app flags as duplicates, you confirm 15. You then check every unflagged invoice and find 10 more duplicates. The flagged list looked convincing, but it missed a substantial part of the problem.
What to do
To check the app, review unflagged invoices too; that's where you find its misses. Here, 15 found plus 10 missed makes 25 duplicates altogether. Precision is 15 of 20 flags, or 75%; recall is 15 of 25 duplicates, or 60%.
Easy to mix up
| Precision and recall | Accuracy |
|---|---|
| Precision checks what was selected; recall checks what was found and missed | Every decision, including leaving unwanted items out of the results |
Check yourself
You test two settings on the same customer messages. One flags 18 real delivery complaints and 12 false alarms. The other flags the same 18 complaints plus 2 false alarms. What changed?
Go deeperF1 combines precision and recall without deciding which error costs more
In his 1979 book, “Information Retrieval,” C. J. van Rijsbergen defines precision and recall for retrieved documents and discusses evaluation measures from a 1966 Cranfield study.
Google's Machine Learning Crash Course describes F1 as the harmonic mean of precision and recall. When their sum is above zero, multiply the two rates, double the result, and divide by their sum. For 80% precision and 50% recall, F1 is about 62%. A low value for either rate pulls this average down. F1 doesn't decide which mistake costs you more.
Sources
- Classification: Accuracy, recall, precision, and related metrics, Google Machine Learning Crash Course
- Information Retrieval, Chapter 7: Evaluation, C. J. van Rijsbergen