Meli AI Mentoring

4.1

They choose models by measured results

Chapter Editorial Status: Early Draft

Chapter Editorial Status

Agent instructionsPaid book

Treat each model setting as a configuration to test

Current AI products can combine a model family, reasoning-effort setting, serving-speed option, search or tool access, and plan-specific limits. Each combination is a configuration. Its quality, latency, and cost depend on the provider, workload, and current release.

Write the quality target before comparing configurations. The same criteria must score every output. This makes your work the basis of the comparison.

This evidence-based assignment has a name.

People already route work across available models. Anthropic's 2026 Economic Index found different model-selection patterns across task categories and task value. Your own evaluation adds the piece that study leaves task-specific: measured quality on the work you perform.

Build an evaluation set from representative cases

A representative evaluation set includes the situations your route will face in practice. Sample routine inputs, difficult inputs, edge cases, and costly failure modes. Refresh the set when the work changes or a new failure appears.

Five task families provide a useful starting inventory: quick transformations, everyday professional work, high-stakes reasoning, current factual research, and file-and-action work. Give each recurring task its own quality target and representative examples.

Comparison

Task family	Quality target	What to measure
Quick transformations	Complete coverage, source fidelity, and exact requested format	Pass rate, elapsed time, and cost across common and messy inputs
Everyday professional work	Accurate, clear, actionable work in the intended voice	Rubric score, revision count, elapsed time, and cost
High-stakes reasoning	Correct evidence, explicit tradeoffs, uncertainty handling, and reviewable reasoning	Expert review, factual checks, failure severity, elapsed time, and cost
Current factual research	Current sources, accurate citations, complete coverage, and traceable claims	Source checks, citation accuracy, elapsed time, and cost
File and action work	Correct changes, preserved constraints, completed checks, and usable output	Task completion, verification results, elapsed time, and cost

Set a quality target. Test representative examples. Compare quality, speed, and cost. Use the fastest, least expensive model that consistently meets the target.

Test current configurations on your own work

Model names, settings, prices, and serving behavior change. Date every comparison and record the full configuration. Re-run the evaluation after a material model update, a price change, or a new failure in the recurring task.

Run every representative example on every configuration in the comparison. Score outputs against the same rubric, record pass or fail, measure elapsed time, and estimate cost. Repeat examples when normal output variation could change the decision.

Save the result as a dated selection reference. Include the task, quality target, evaluation examples, tested configurations, pass rates, latency, cost, and chosen route. That record gives you a starting point for the next comparison.

Build a model-selection reference from one tested evaluation

Hand-drawn evaluation diagram showing representative cases tested on Config A and Config B, with pass, fail, time, and cost recorded in a dated routing notebook. — Test representative cases on each configuration. Record pass or fail, time, and cost, then save the selected route with the date.

Turn the evaluation into a reference you keep. The prompt below helps you define the target, prepare representative examples, record measured results, and choose the fastest, least expensive configuration that consistently passes.

Build your personal model-selection referencePaid book · Claude helps you define a quality target, prepare representative examples, and record quality, speed, and cost.

References

1 source

1
Economic Index: Learning Curves.
Anthropic · 2026
Paid Claude.ai users send 55% of Computer and Mathematical tasks to Opus versus 45% of educational tasks. Users working on higher-value tasks (as measured by the typical wage for that type of work) are significantly more likely to choose Opus. The report separately finds that higher-tenure users show higher success rates.
The study documents model-selection patterns across observed tasks. Representative evaluations can measure quality, speed, and cost for your own tasks.