Test-time compute
- Reasoning
- Standard term
Test-time compute is the computing an AI uses while producing or choosing an answer, as opposed to the computing used to train it. More of it can mean longer thinking, several drafts, or extra checks.
Extra thinking gives the model room to try an approach, catch a mistake, and back up, which helps on problems with many dependent steps. On simple tasks it adds waiting for the same answer.
Where you’ll meet it
This month's bill for the AI assistant your team uses at work is double last month's. The usage page shows nearly all of it came from a "high effort" setting someone turned on for every request, including one-line replies to email.
What to do
Run a few past jobs at both settings, and keep the high setting only for the kinds of work where its answers came out better, such as reconciling two spreadsheets or tracking down a bug. Then set the default back to a lower effort for everything else.
Easy to mix up
Pretraining is paid for once, by the lab that built the model, before release. Test-time compute is paid for with every request, in waiting time and usage.
Check yourself
You have two requests for the same assistant and can turn its effort setting up for only one. Which is more likely to come back better for it?
Go deeperAnswer-time computing can sometimes stand in for a bigger model
The idea, also called “inference-time compute,” predates reasoning models. Self-consistency (Wang and colleagues, 2022) samples several answers and keeps the most common. In 2024 OpenAI reported its reasoning models grew more accurate with more thinking time, and Snell and colleagues (2024) found extra answer-time computing could match a much larger model on problems the smaller one sometimes solved.
Sampling many answers helps most when something can score them, such as tests that pass or a result that checks out, so the system keeps the best one. Without a scorer, a majority vote gains less with each extra sample and soon levels off. Product settings such as "reasoning effort" or a thinking budget control only part of this process.