Confidence calibration
- Checking AI's work
- Standard term
Confidence calibration checks how well the chances an AI gives match what happens. If it assigns 100 deliveries an 80% chance of arriving on time, good calibration means about 80 arrive on time.
An unlikely outcome can still happen, so one surprising result doesn't settle whether a forecast was reliable. You need many predictions grouped by the chances the AI gave them, with their outcomes checked afterward.
Where you’ll meet it
At the end of the quarter, you compare purchases with your sales app's lead forecasts. A group labeled “90% likely to buy” bought far less often than nine in ten.
What to do
Group leads given similar chances of buying and compare each group's purchase rate with its forecast for the same period. Use the actual purchase rates to check your revenue plans while you investigate why the app's forecasts were too high.
Easy to mix up
| Confidence calibration | Example-led calibration |
|---|---|
| Predicted chances and how often the outcomes happen | AI answers and examples of the result you want |
Check yourself
One system's “90% sure” answers are correct 70 times out of 100. Another system's “60% sure” answers are correct 61 times out of 100. Which confidence label better matches its observed record?
Go deeperReliability diagrams show mismatches, and question format can affect confidence
Guo and colleagues' 2017 paper, “On Calibration of Modern Neural Networks,” describes reliability diagrams: charts comparing stated confidence with how often answers were correct. Perfect agreement follows a diagonal line. Points away from that line show which confidence ranges overstate or understate the observed rate.
Kadavath and colleagues' 2022 paper, “Language Models (Mostly) Know What They Know,” found well-calibrated answers on multiple-choice and true-or-false tasks with suitable question formats. Models trained further using reward signals, a method called reinforcement learning, initially gave probabilities that matched results poorly. A “temperature adjustment” made the probabilities less extreme and largely corrected the mismatch on the tasks studied.
OpenAI's 2023 technical report found close agreement between a model's confidence and correctness on part of a general-knowledge test. After further training to shape its responses, agreement weakened. That result applies to the test and model studied. An assistant writing “90% sure” in your chat needs its own record of checked answers before you can assess that claim.
Sources
- On Calibration of Modern Neural Networks, Guo et al., ICML 2017
- Language Models (Mostly) Know What They Know, Kadavath et al., arXiv
- GPT-4 Technical Report, OpenAI, arXiv