A result supports trust under the conditions you tested. Suppose your assistant summarizes a clear client-call transcript accurately. After a model update, you begin supplying transcripts with several speakers talking over one another, and a summary omits a decision. The earlier success still happened, but it does not settle whether the changed setup handles the harder input. The agent saves the conditions with the result so you know what to retest.
Becoming an AI Power User
Becoming an AI Power User
147 min leftPage 146 of 226 (est.)
4.5Calibration and Evaluation
They maintain calibration by retesting when tasks, tools, or requirements change
Developing
Keep the result and the conditions together. A table can record the task, tested configuration, evidence, and verification method. Retest triggers identify changes that call for a new check, and a recurring review helps you find gaps you have not otherwise noticed. Ask an assistant to maintain the record and propose retests from logged failures.
Record what you tested and what each result supports
A broad label such as 'reliable' leaves important details unstated. Which task? Which inputs? How many checked results? A grid records that information in one row per recurring task. Dates help you locate the evidence, and specific cases show how much the rating supports.
A morning brief draws on several tasks: reading the calendar, identifying unfinished commitments, and drafting preparation notes. Have the agent record each separately when it has different evidence or consequences. For example: 'Calendar times: six checked briefs, all times matched the supplied calendar; last checked September 4; retest after calendar-access changes.' That illustrative row supports a limited claim about those cases, not certainty about every future brief.
For the preparation note, the record might say: 'The assistant called a routine meeting urgent without supporting notes; priority accuracy remains uncertain.' Ask an evaluator to compare the brief with the calendar and priority notes, then save the mismatches it identifies. Check a sample of both flagged and unflagged results so the evaluator's blind spots do not silently become your reliability rating.
Checking methods should target errors that matter
Choose the check from the evidence, the consequences, and what could detect the error. A familiar formatting task may warrant a quick glance; a budget total can be recalculated directly. A research conclusion may require a reviewer with access to the sources and relevant expertise. More review is useful when it catches consequential mistakes at an acceptable cost. Review effort alone does not establish quality.
Consider the checks for a morning brief. If a reformatting step has preserved the task list across representative examples, a quick comparison may be enough for that low-stakes use. A preparation note needs comparison with the meeting's purpose and your priority notes. A budget figure that determines a decision deserves a direct source check or recalculation, even after many accurate briefs. The check should address the particular error that could matter. Calling a task low-, medium-, or high-stakes helps only alongside an explanation of the consequence and how the check could detect it.
Use the grid to explain why a check is appropriate. A verified pattern across representative low-stakes cases may support lighter review. A new input type or unexplained failure may call for closer checks. For a consequential figure, you may still want a direct calculation or source check even after many successful runs.
Use relevant changes to choose the next retests
A model update is one retest trigger. Changed instructions, data access, tools, input types, or consequences can be just as relevant. Start with the tasks those changes could affect and with failures you would find costly. A new capability is also a reason to revisit something you previously could not do.
Past failures, established successes, and tasks that depend on a specific behavior all supply useful test candidates. A previously failed task can reveal new capability. Repeating a previously passing case can reveal a : the case fails after a change. A formatting-dependent task can reveal a change that would disrupt the next step. Choose from these according to what changed; they are not a ranking of which tasks will shift.
Schedule a review to catch gaps between change-triggered tests
A recurring review can help you find missing evidence between known changes. Open the grid, inspect recent failures, and ask which judgments still fit the work. A monthly review may be a convenient starting habit for a frequently used brief. Choose a different interval when use, consequences, or change frequency warrants it, and retest important changes without waiting for the appointment.
Use a review to explore possibilities as well as failures. You might test an imaginative new use, clarify what you want from a report, compare a different configuration, or improve a saved . Save the specific improvement and test whether it helps on later cases. Curiosity, strategy, evaluation, motivation, and routines contribute together; a calendar interval does not guarantee progress.
Build your grid and choose when to review it
Build your trust calibration grid and recalibration schedule
With your test records, Claude can help organize checked results and gaps, then propose retest triggers and an optional review schedule suited to your tasks.
- Copy the promptUse the copy button above the box.
- Paste it in Claude or CodexUse the chat or project where you want the assistant to do the work.
- Answer the questionsGive clear answers so it understands what you want made.
- Test it to see if it worksTry the output and refine until it does what you need.
Problem: My expectations about AI may no longer fit the tasks and conditions behind them.
Intent: I want a dated record of what I have checked, what remains uncertain, and what deserves retesting.
Ask about recurring AI tasks where quality matters, the configuration used for each, what a mistake would affect, and how I currently check the result. Ask me to paste or attach relevant test logs, outputs with their source material, or notes from checks I completed. If the evidence is unavailable, ask what I can provide and leave the gap explicit.
Use my actual records and answers. Distinguish a result checked against evidence from a general impression or a remembered success. Ask for dates where they are missing; do not invent them.
Please:
1. Draft a trust calibration grid with these columns: Task, Configuration, Reliability (high/medium/low/unknown), Known Failure Modes, Current Checking Method, Evidence, Retest Trigger, Last Tested, and Recommended Adjustment (increase/maintain/decrease checking, or gather evidence before deciding).
2. For each reliability label, explain exactly which checked cases and conditions support it. Mark reliability unknown when the evidence is missing or does not fit the current conditions. Treat example counts as descriptions of evidence, not proof of a stable failure rate.
3. Compare the current checking method with the possible consequences and the errors it could detect. Explain any proposed adjustment using my records. Distinguish a tested AI judge from an evaluator whose decisions still need checking.
4. Prioritize retests by relevant changes, consequences, observed failures, and missing evidence. Specify the input, expected behavior, and check for each proposed test. Include past successes as well as failures when the change could affect them.
5. Suggest an optional review interval from how often the task runs, what changes, and what an error would cost. A date alone does not establish that a result has expired. Do not create calendar events.
6. Return a recalibration checklist: what to retest, what evidence to update, and which changes should prompt a check before the next scheduled review.
7. Suggest a different configuration only when a comparison could address an observed weakness or uncertainty. Explain what the test would reveal; leave it untested until I provide results.
Scope: structure the grid and propose checks. Do not carry out my recurring tasks or claim current model behavior without checked evidence. Return the draft here, show the records and stated preferences behind it, and wait for my confirmation before saving it as a reusable personal reference.
References
1 source- 1
Anthropic · 2026
Anthropic's January 2026 engineering guidance describes repeated trials, representative tasks, checks on outcomes, and evaluation environments that resemble actual use.
Save the tested conditions and expected behavior so comparisons after changes remain meaningful.
This is practitioner guidance from agent development. It does not establish a universal review interval or make a small personal test set a reliability guarantee.