LLM-as-judge (AI grader)
- Checking AI's work
- Standard term
An LLM-as-judge uses a large language model, the AI inside many chat apps, to score or compare AI answers. You describe what a good answer must do. The judge's assessments also need checking.
The judge reads an answer and instructions for grading it, then writes its assessment. It may favor an answer shown first or a longer one, even when it is less clear or accurate. It can also accept a wrong answer if it lacks the knowledge or source documents needed to check it.
Where you’ll meet it
A teammate uses an AI grader to score 500 product descriptions for brand voice. Nearly every description receives an 8 or 9, including several you'd send back for rewriting. The grading instruction says only “Does this sound like us?”
What to do
Rate some descriptions yourself, including ones you'd accept and ones you'd send back. Give the judge those ratings and explain why each description does or doesn't sound like your brand. Then rate a different batch and compare your ratings with the judge's.
Easy to mix up
| LLM-as-judge (AI grader) | Self-critique |
|---|---|
| A score or comparison in a separate grading step | Feedback to help revise the model's own draft |
Check yourself
An AI judge compares Alex's and Sam's welcome emails. First it sees Alex's above Sam's; then you reverse them. Which outcome gives you more reason to trust its choice between the drafts?
Go deeperGiving a judge correct math answers reduced grading mistakes in a 2023 experiment
In Zheng and colleagues' 2023 study, one judge accepted incorrect math answers in 14 of 20 cases with default grading instructions. When also given correct answers for comparison, it accepted 3 of 20. These historical results concern a particular setup and show how giving a judge a correct answer can affect its reliability.
Anthropic's January 2026 agent-evaluation article identifies another tradeoff: model-based graders can give different assessments across repeated attempts and cost more than checks written in code. When a requirement has an exact, checkable answer, a fixed rule can check it without asking a model to make that judgment.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al., NeurIPS 2023
- Evaluation best practices, OpenAI API docs
- Demystifying evals for AI agents, Anthropic Engineering