Quality ratchet
- Agents
- Industry term
A quality ratchet is the practice of asking for a better version again and again, keeping the best result so far as the starting point for each new round, until more rounds stop producing improvements.
An AI's first good answer often has room to improve. Shown a result and asked to beat it, a model can find a clearer structure, a stronger example, or a missed case. It can also drop a fact on the way, so reading each new version beside your best one shows whether the round helped.
Where you’ll meet it
You asked an AI to write the welcome email for an online class you're starting. The draft is fine: friendly, clear about the schedule, the right length. You could send it now.
What to do
Ask for a better one anyway, and say what better means, such as first steps a nervous beginner could follow. Keep the stronger version and ask again from it. When new versions stop improving anything your students would notice, send the best one you have.
Easy to mix up
With good enough, you stop asking once the result meets your standard. A quality ratchet keeps asking for better until more rounds stop producing gains.
Check yourself
Round four of your conference talk outline has a sharper opening than round three, but its ending drops your main takeaway, so overall it's weaker. Which version should round five improve on?
Go deeperSoftware teams use ratchets to keep a number from falling
In software, a ratchet can be a build check that lets a measure such as test coverage rise and blocks a change that lowers it. With a draft, keeping the stronger version aims for the same one-way improvement, but judging quality is less mechanical and the judgment can be wrong.
Once gains stop, more rounds may only exchange one acceptable version for another. An AI judge can mistake greater length or polish for improvement, so compare the versions against the reader's needs. If you cannot name a useful gain, keep the best supported version rather than accepting novelty alone.
Sources
- Codecov Status Checks, Codecov documentation
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv