Becoming an AI Power User

4.4Calibration and Evaluation

They build evaluation workflows with rubrics, examples, and judges to scale quality standards

Developing

A tested judge can apply your criteria across recurring work

As you delegate more work to AI, you may have more outputs than you can inspect individually. If quality checks depend entirely on your reading every output, unreviewed results have not received those checks. To delegate part of the review, you need to write down the criteria you apply and measure whether another evaluator can apply them to new work.

An is a model asked to evaluate an output against written criteria and evidence. Give the reviewing session your criteria, the output, and the evidence needed to check it. It can run the same written procedure across many outputs, but its verdicts are still fallible and may vary. This chapter shows how to measure those errors, limit the judge to the scope you tested, and send uncertain or consequential cases to a qualified reviewer.

A judge is revised using development cases, locked, then assessed on untouched cases for false accepts, false rejects and abstentions. — Development cases may loop through revision. Final test cases stay outside this loop and test the locked judge under a procedure fixed in advance.

Choose a check that can answer the quality question. A calculation or exact comparison can check a total or a scheduled time. Deciding whether preparation notes address the reader's priorities requires evidence about those priorities and a judgment about relevance. Use a and examples to communicate that judgment to an AI reviewer, then test whether its verdicts match decisions you have checked.

Suppose an assistant prepares a morning brief from your calendar and task list. A useful brief preserves the scheduled times and draws attention to commitments that need preparation. A judge can compare the brief with those records and your written preferences. Test whether it catches omitted meetings, invented tasks, and unsupported priorities on examples it has not seen before.

need explicit criteria and scored examples

Give the judge the reasons behind your verdicts. If a brief feels poorly prioritized, explain which commitment it buried and why that commitment needed attention. Pair the output with the calendar or priority notes that justify your criticism. The assistant can help you articulate the criterion; then check whether it captures the distinction you intended. An unexplained label such as 'weak' leaves the candidate judge guessing what to apply elsewhere.

Write the criteria, show passing and failing outputs, and include close cases. A states the criteria; labeled examples show how you apply them. For a morning brief, an accurate but poorly prioritized summary can reveal a different problem from an invented meeting. Compare candidate judge instructions on development cases before trusting the configuration on new work.

criteria need evidence the judge can inspect

A states what quality requires for a particular task. For a morning brief, every meeting should match the supplied calendar, and anything marked as needing preparation should appear prominently. The judge needs the calendar and your priority notes to assess those criteria. It cannot establish whether a meeting exists from the brief alone.

Reuse an existing when it describes the same task and still fits your requirements. For a morning brief, accuracy criteria may carry over, while priorities need the current calendar and your notes about what requires preparation. Check a borrowed criterion before adopting it. If you have no , ask the assistant to draft one from outputs you have judged, then correct any requirement your examples do not support.

Accepted examples need an explanation of why they pass

A is an output you would accept without changes, paired with the reason it passes. Choose representative examples rather than a fixed number. They give the candidate judge concrete decisions to inspect, but a judge may match their familiar wording while missing the criterion on new outputs. Compare the configuration on different development cases before deciding whether the examples help.

Rejected examples show which failures to catch

A rejected example pairs weak output with the reason it fails. A smooth brief that invents two meetings documents an accuracy failure, even when the prose reads well. Choose failures the judge is likely to encounter, then test whether including those examples improves its decisions on different development cases.

Include boundary cases to measure close decisions

A boundary case is an output close to the pass or fail line, paired with your verdict and reason. Suppose a morning brief lists every meeting accurately but places a negotiation requiring preparation near the bottom. If the supplied priority notes require preparation-heavy commitments at the top, that brief fails the ordering criterion. A routine meeting in the same position may pass. Include the notes that explain the different verdicts so the judge can evaluate the close case from the available evidence.

Tune on development cases, then lock the judge before the final test

A fresh judge is a guess about your standard, not proof that it can handle new work. Separate the cases you use to improve it from the cases that will decide whether it is ready. Development cases are the examples you may inspect repeatedly while you revise the , prompt, model, or tools. Repeated revisions can make the judge match familiar cases without applying the criteria to new ones. The final test set stays untouched during that work. It should contain both known failures the judge must catch and clean outputs it must leave alone, including close cases that resemble the work it will see.

Use development cases to compare candidate configurations and investigate disagreements. Before the final test, the agent freezes the prompt, , examples, model, tools, and test procedure, including any repeated trials. It runs the locked judge in a fresh context with the case and the evidence it would receive in use. It keeps that case's expected verdict and explanation outside the judge's context and presents them beside the recorded verdict afterward for your comparison. It completes the fixed procedure without tuning between results. If the findings lead you to revise the judge, the agent retains those cases for development and collects fresh cases for your next readiness decision.

A judge and its generator can share the same blind spot

A different model is a candidate judge to test because it changes the evaluator, but its name does not prove that its errors are independent. Model families can still miss the same cases. Compare proposed judges on the labeled development cases. Test only the selected, locked configuration on the untouched final set.

You might compare another model, a fresh context with clearer criteria, a rule-based check, or a qualified reviewer. Measure which candidate catches consequential failures without inventing problems in correct work. A different provider can supply a useful comparison, but provider identity alone does not establish independent errors. UNESCO describes complex recitation techniques used to preserve the Vedic oral tradition; Radha Kumud Mookerji's historical account includes overlapping word pairs. Suppose the English sentence 'Bring fresh water tomorrow' becomes 'Bring water tomorrow'. Comparing the shortened sentence with the separately learned pair 'bring fresh' exposes the omission because the pair retains the missing word, even though the shortened sentence still makes sense. Pairs generated from the shortened sentence would repeat its omission, and learned forms can still share mistakes. For an , tests must establish which generator errors it catches, which it repeats, and which correct outputs it wrongly rejects.

Automate only the decisions the untouched test supports

The agent counts the mistakes with different consequences separately. A false accept approves work that should fail; a false reject rejects work that should pass. An abstention reports that the judge cannot decide. The agent records the number of each, the relevant passing and failing case counts, and the fraction of all cases receiving a decision. That fraction is decision coverage.

A small test can reveal different kinds of error

Suppose a final set contains ten faulty briefs and ten acceptable ones. The judge approves two faulty briefs and rejects the other eight. It rejects one acceptable brief, approves eight, and abstains on one. The false-accept rate is 2 out of 10 faulty briefs, or 20%; the false-reject rate is 1 out of 10 acceptable briefs, or 10%. It decided on 19 of 20 cases, so coverage is 95%.

Those denominators describe the tested cases. With only ten cases in each group, one additional false accept or false reject raises the corresponding rate by ten percentage points. These invented numbers demonstrate the calculation, not a suitable sample size or permission to automate. Choose the evidence requirement from the consequences and your tolerance for uncertainty.

Use the final results to decide whether a pilot with review is warranted. Consequential unattended decisions need a statistically justified error estimate from representative cases. Choosing a minimum case count does not supply that justification. A qualified evaluator can help determine how much evidence the decision requires and how to account for uncertainty. Until that question is resolved, retain review. Limit any use to the tested task, inputs, and actions, with abstention or escalation for missing evidence or unfamiliar cases. After a relevant change or new failure, retain the old cases for development and checks, then test a revised judge on fresh final cases.

Build a judge without tuning on its final test

Build and test a judge for a recurring workflow

Using your labeled examples, Claude can help you tune on development cases, lock one judge, and prepare a separate final test with error limits set in advance.

Copy the promptUse the copy button above the box.
Paste it in Claude or CodexUse the chat or project where you want the assistant to do the work.
Answer the questionsGive clear answers so it understands what you want made.
Test it to see if it worksTry the output and refine until it does what you need.

Problem: I need to find out whether an AI judge can apply my standards to new work. Intent: I want to build and test a judge for a recurring task while keeping its final test separate from every tuning decision. Ask about the task, the decisions the judge might make, the consequences of accepting faulty work or rejecting acceptable work, and the records or sources needed to check the output. Ask for my actual passing, failing, and close examples, with my verdicts and reasons. Use only examples and labels I supply or confirm. Do not invent outputs, labels, preferences, scores, or test results. Tell me to reserve a separate final set before tuning begins. Keep those final cases and their labels outside this conversation. They should include representative faulty outputs, acceptable outputs, and close cases. Keep variations of the same source case in the same group so the final set tests different underlying cases. We may use development cases to revise the judge. Before the final test, help me state the acceptable false-accept and false-reject limits, when the judge must abstain or escalate, and how much decision coverage would make it useful. For consequential unattended decisions, identify the statistical method and assumptions needed to justify the number and variety of test cases. Flag an unjustified evidence requirement as unresolved; do not invent a sufficient sample size. Please: 1. Check whether the proposed criteria can be assessed from the available evidence. Use exact calculations or rule-based checks for requirements they can establish. Reserve the AI judge for criteria that require judgment. 2. Draft a rubric and self-contained judge prompt from development material only. Include the needed sources, the reasons examples pass or fail, and an abstain result for missing evidence or cases outside scope. 3. Check which candidate configurations can be run with the tools available. Give each candidate a fresh context with its instructions and the case evidence, withholding the expected verdict for the case being scored. Keep scored cases separate from examples embedded in the judge prompt. If you cannot run a configuration, give me copyable instructions and wait for its actual results. 4. Compare development verdicts with my labels. Record false accepts, false rejects, and abstentions. Help me investigate disagreements and revise the candidate until I select a configuration. 5. Return a lock record with the selected prompt, rubric, examples, model and settings, tools, version, and fixed test procedure, including any repeated trials. 6. Explain how I can run that locked judge in a fresh context on the reserved final cases without revealing their expected verdicts or explanations. Keep the procedure unchanged throughout the final run. 7. After I provide the recorded verdicts and labels, report false accepts divided by the number of faulty cases, false rejects divided by the number of acceptable cases, abstentions, and decision coverage: the share of all cases receiving a verdict. Include counts, denominators, and sample size, and distinguish distinct cases from repeated trials. Compare the results with the limits set in advance and explain the remaining uncertainty. 8. If a final result informs a revision, retain those cases as development evidence and require fresh untouched cases for the next readiness decision. Limit any proposed use to what the tested scope and evidence support. Scope: build, tune, record the chosen configuration, and prepare the test. Interpret actual results when I supply them. Do not complete the original task or declare the judge ready before the final evidence supports that conclusion. Return the rubric and records here; before saving them as a reusable personal standard, show what they preserve from my material and wait for my confirmation.

References

1 source

1
Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges View source
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle, Miguel R. D. Rodrigues · 2026 · ICLR 2026
The paper treats an LLM judge as an imperfect measurement device and estimates its true-positive and false-positive behavior on human-labeled calibration data instead of treating raw judge labels as ground truth.
A useful judge needs error-specific measurement against trusted labels before its verdicts support an automated decision.
The paper develops a statistical certification framework. This chapter applies its simpler operational lesson to a personal workflow and does not claim that a small example set supplies the paper's guarantees.