Becoming an AI Power User

4.1Calibration and Evaluation

They choose models by measured results

Developing

Choose a model that meets your quality target and fits the task

Start by writing what a passing result must do. A strategy brief might need accurate evidence, explicit tradeoffs, and a usable recommendation. A meeting-note cleanup might need complete coverage, exact names, and the requested format. Those criteria form your quality target.

Next, choose representative examples from your recurring tasks. Include a common case, a difficult case, and an that has caused trouble before. Run the same examples through each configuration you want to compare.

Score every output against the quality target. Record elapsed time and estimated cost beside the score, including retries and corrections needed before you could use it. Choose among the passing configurations according to the trade-off your task requires. A live conversation may favor quick responses; an overnight report may favor a more careful result.

Treat each model setting as a configuration to test

Current AI products can combine a model family, reasoning-effort setting, serving-speed option, search or tool access, and plan-specific limits. Each combination is a configuration. Its quality, latency, and cost depend on the provider, workload, and current release.

Decide which requirements every passing output must meet. For meeting notes, an invented deadline can be a failure even when the summary is clear and complete. Score qualities such as clarity separately so a strong score cannot conceal a disqualifying error. Score every configuration the same way.

A saved model choice helps with recurring tasks. If your AI tool supports automatic routing, use the tested choice to configure it. Otherwise, use the reference to select the configuration yourself. Access to a model alone does not establish that an assistant can switch to it.

Observed model choices differ by task. Anthropic's March 2026 Economic Index found different selection patterns across kinds of work. Those choices show what users selected; your own comparison tests whether a configuration produces the result you need.

Build an evaluation set from representative cases

A representative evaluation set includes the situations your route will face in practice. Sample routine inputs, difficult inputs, edge cases, and costly failure modes. Refresh the set when the work changes or a new failure appears.

Five task families provide a useful starting inventory: quick transformations, everyday professional work, high-stakes reasoning, current factual research, and file-and-action work. Give each recurring task its own quality target and representative examples.

Task family	Quality target	What to measure
Quick transformations	Complete coverage, source fidelity, and exact requested format	Pass rate, elapsed time, and cost across common and messy inputs
Everyday professional work	Accurate, clear, actionable work in the intended voice	score, revision count, elapsed time, and cost
High-stakes reasoning	Correct evidence, explicit tradeoffs, uncertainty handling, and reviewable reasoning	Expert review, factual checks, failure severity, elapsed time, and cost
Current factual research	Current sources, accurate citations, complete coverage, and traceable claims	Source checks, citation accuracy, elapsed time, and cost
File and action work	Correct changes, preserved constraints, completed checks, and usable output	Task completion, verification results, elapsed time, and cost

A few examples are a starting comparison. They can expose an obvious mismatch, but they cannot establish a dependable failure rate. Add cases that resemble your actual inputs and repeat important cases when output variation could change your choice. Increase the evidence you require as the consequences of an error grow.

Compare model configurations using the same tasks and

Model names, settings, prices, and serving behavior change. Date every comparison and record the full configuration. Re-run the evaluation after a material model update, a price change, or a new failure in the recurring task.

Run every representative example on every configuration in the comparison. Score outputs against the same , record pass or fail, measure elapsed time, and estimate cost. Repeat examples when normal output variation could change the decision.

Save the result as a dated selection reference. Include the task, quality target, evaluation examples, tested configurations, pass rates, latency, cost, and chosen route. That record gives you a starting point for the next comparison.

Build a model-selection reference from one tested evaluation

The same ordinary, difficult and edge cases enter two configuration lanes, with quality, time, cost and test date left for comparison. — Test representative cases on each configuration. Record pass or fail, time, and cost, then save the selected route with the date.

Save enough evidence to revisit the choice. The prompt below helps you define the target, prepare examples, and record the results. Include the trade-off you chose so a later comparison can test whether the reason still holds.

Build your personal model-selection reference

Claude helps you define a quality target, prepare representative examples, and record quality, speed, and cost.

Copy the promptUse the copy button above the box.
Paste it in Claude or CodexMake sure your active project is enabled.
Answer the questionsGive clear answers so it understands what you want made.
Test it to see if it worksTry the output and refine until it does what you need.

Problem: I need evidence for choosing an AI configuration for recurring work. Intent: I want to compare configurations on my own tasks and keep a dated reference explaining the choice. Ask me about the recurring task, what an acceptable result must do, and my quality, time, and budget priorities. Use my answers before asking for further details. Ask me to paste or attach representative inputs and any outputs I have already checked. Include ordinary work, known difficulties, and relevant edge cases; identify gaps without inventing my work or test results. Ask which configurations I can access. For each, record the model, reasoning or effort setting, serving-speed option if available, enabled tools, plan, and test date. Use dated price information I supply or verify it from the provider. Leave unknown prices and unavailable settings explicit. Please: 1. Draft a short scoring rubric. Distinguish requirements every passing output must meet from qualities we can compare among passing outputs. 2. Propose an evaluation set from my material and explain which requirement each case tests. 3. Give me the same task instructions to use with each configuration and a results table with columns: Case, Configuration, Quality score, Pass or fail, Elapsed time, Estimated cost, Retries and corrections, and Notes. Count the effort needed before the output is usable. Explain what each cost estimate includes. 4. Wait for actual results. Use only results I provide, and mark untested fields as "needs testing." If I can access only one configuration, evaluate it against the target and leave the comparison open. 5. Compare the observed results with my priorities. Recommend a passing option when the evidence supports a choice. If none passes, or the evidence cannot distinguish them, explain the remaining gap. Treat small samples as exploratory and suggest further cases or repeated trials only where they could change the decision. 6. Return a dated selection reference with the task, target, examples, tested configurations, results, proposed route, trade-offs, evidence limits, and next review trigger. Return the reference here for me to review. Before saving it as a reusable personal reference, show the actual results and preferences it records, ask where I want it stored, and wait for my confirmation.

References

1 source

1
Anthropic Economic Index report: Learning curves View source
Anthropic · 2026
In the February 5–12, 2026 sample, paid Claude.ai users selected Opus for 55% of Computer and Mathematical tasks and 45% of Educational tasks. Selection also correlated with task value, estimated from typical US occupational wages.
The study documents model-selection patterns across observed tasks. Representative evaluations can measure quality, speed, and cost for your own tasks.
The report observes one provider's users and model choices. It does not establish that those choices caused better results or that occupational wages measure the value of your particular task.