Self-consistency
- Reasoning
- Research term
Self-consistency is a technique where an AI works the same problem several separate times and keeps the final answer that comes up most often, like a majority vote.
A model picks each word with some randomness, so separate runs on one problem take different routes. When correct routes tend to land on the same answer and mistakes scatter across many, the most common answer is right more often than a single run.
Where you’ll meet it
You ask your assistant to estimate next month's electric bill under a new time-of-use plan, working it out three separate times. Two estimates agree, and the third comes out twenty dollars higher.
What to do
Ask for the rates and hours behind each estimate, and find the step where the higher one parts from the other two, since one side went wrong there. Your utility's rate sheet settles any rate they disagree on.
Easy to mix up
In Best-of-N, a single run's answer wins whenever the scorer rates it highest. In self-consistency, the most common answer wins, so a one-off answer wins only in a tie.
Check yourself
You give a model a project's start date and deadline and ask, in five separate runs, how many working days the project has. All five say 15. Which mistake does that agreement make unlikely?
Go deeperVoting needs answers that can be counted
Xuezhi Wang and colleagues introduced self-consistency in 2022 for chain-of-thought prompting, where a model writes out its reasoning before answering. In place of the single most likely reasoning path, it samples many and tallies their final answers, setting the reasoning aside. It runs while the model answers and needs no training.
Numbers and multiple-choice letters tally cleanly, often after normalizing so “$1,200” and “1200” count as one. Open-ended writing has nothing to tally, so later variants ask a model to pick the response most consistent with the others. Five samples take about five times the computing of one answer.