Becoming an AI Power User

5.6Infrastructure and Learning Systems

They build learning loops that test and carry improvements into future work

Developing

Test recurring corrections in the setup that produces the work

A client update buries a decision, so you rewrite the opening. If the same problem recurs, ask what in the setup produces it: a missing requirement, a weak example, or a reviewer that checks style but ignores usefulness. A connects that observation to a tested change in future work.

A buried decision might prompt an experiment with a shorter opening, a stronger example, or a different review question. Curiosity and play suggest alternatives; your goals and strategy help you choose which to try; evaluation shows what helped; and a routine for saving tested changes lets later work use the result. Record a promising change as a candidate until a comparison supports keeping it.

Setting up examples, criteria, and a review routine takes effort. Recurring work may justify that setup when repeated errors or preparation costs matter. A consequential one-off task can also deserve careful evaluation, even if it needs no ongoing feedback routine. Choose the amount of setup from the task's consequences and the future uses you reasonably expect.

A weekly client update gives the loop a concrete job: explain progress, risks, and decisions to a non-technical client. If a draft buries a decision, revise the draft and investigate whether the recurring instructions, evidence, or checks need to change. Save a proposed improvement, compare it with the previous setup, and keep it when it helps across relevant cases. Curiosity supplies new possibilities to try; evaluation helps you decide which ones to carry forward.

An observed buried decision suggests a saved workflow revision; previous and candidate versions are compared on shared examples before keeping a supported change. — A tested workflow change can carry a useful lesson into later runs. Keep a candidate only when the checks support it; retaining the previous version is also a valid result.

Generate, Compare, Specify, Judge, Learn, Update describes the sequence used here. Start with known requirements, try meaningful variations, and compare them to refine criteria you cannot yet express. Test a judge against reviewed examples, investigate disagreements, and propose changes to the workflow. Compare each candidate with the previous setup before keeping it. A loop can end with the original version retained and a clearer account of what failed.

Recurring failures can originate in the saved setup

Saved prompts and workspaces preserve requirements; a tests whether changing them would improve results. For the client update, a decision could be buried because the drafting instructions omit it or because the review checks only tone. Compare the draft and review with the source notes to locate the gap before changing either. Reuse supplies the starting setup, and evaluation helps decide what deserves revision.

A can improve a method or question what the method is trying to achieve. Chris Argyris called that second kind : reconsidering the governing assumptions or goals. In our example, changing the template to put decisions first improves how you write the update. Asking whether a weekly recap serves the client at all, and choosing decision-triggered briefings instead, questions its purpose. Saving a change in a makes it persistent; persistence alone does not make it double-loop learning. Argyris, Double Loop Learning in Organizations (opens in a new tab).

The loop connects an observation about the result with a proposed change to how future work is done.

Comparing outputs helps you refine criteria you cannot yet express

You may know some requirements before drafting: the audience, the source of facts, and what decision the update supports. Other qualities, such as tone or emphasis, may be harder to express. Start with the requirements you know and use comparison to discover what is missing. A strong and a weak version can give a vague preference a concrete reference.

Ask for several meaningfully different versions using the same project facts. Compare which one helps the client understand the decision and why. If one buries an important uncertainty, name that failure and add a criterion for it. Keep factual and ethical requirements fixed while exploring presentation; a version you prefer stylistically can still be wrong.

Generation is a way to see your own taste, not only to produce work

There is a reason comparison surfaces criteria that introspection misses. Your judgment lives partly as recognition rather than as a stated rule: you know it when you see it, before knowing how to say what 'it' is. Comparing several concrete versions helps identify and name the details behind that recognition. A power user generates partly to map where their preferences sit and partly for the play of it: asking for a wilder version, a shorter one, one in a voice you would never use, because an unexpected attempt sometimes shows you a criterion you did not know you held, and the experiments are genuinely fun to run. Each batch of variations is both a probe and a small experiment, and the difference between two versions is often clearer evidence of a criterion than either version alone.

Comparison helps you articulate preferences and challenge them. Ask why one version serves the reader better, and look for evidence beyond which wording you happen to like. A client may need more context than you do. The resulting criteria can incorporate your intent, audience needs, source requirements, and what the examples reveal.

Discover your evaluation criteria through comparison

Claude drafts contrasting versions from your task and source material, then uses your comparisons to propose a rubric and examples for future evaluation. You check that they reflect your intent and the reader's needs.

Copy the promptUse the copy button above the box.
Paste it in Claude or CodexMake sure your active project is enabled.
Answer the questionsGive clear answers so it understands what you want made.
Test it to see if it worksTry the output and refine until it does what you need.

Problem: Some qualities I want in a recurring output are difficult to explain before I compare examples. Intent: I want to clarify those criteria using my task, source material, and judgments. Ask for the task, intended reader, known requirements, and the source material needed to produce meaningful versions. Use what I provide; ask for missing facts rather than inventing them. Generate labeled versions that explore materially different structures, tones, or emphases. Keep the underlying facts and required constraints consistent across versions. Then wait for me to compare them. I may prefer different parts, reject all of them, or need another variation. Use my comments to explain which choices help or hinder the intended reader. Separate my preferences from factual requirements and criteria whose support is still uncertain. Ask a focused follow-up when my reason for a preference is unclear. Draft a rubric with concrete descriptions of strong, partial, and weak performance. If using a 1-5 scale, explain what each score means for that criterion. Distinguish requirements that must be met from qualities that can trade off. Recommend examples to retain and explain what each demonstrates. The strongest of a weak set may still need revision before it can serve as a gold example. Show the criteria and their supporting comments or passages, and ask me to confirm before saving them as my reusable standards. Return the draft rubric and suggested examples here. Do not infer preferences I have not expressed or save generated examples as writing I authored.

Examples carry your judgment further than adjectives can

Naming a criterion introduces a second problem: words like 'professional,' 'clear,' 'strategic,' 'warm,' and 'sharp' are weak cues. Two careful readers can look at the same client update and disagree on whether it is 'professional,' because the word points at different things in each of their heads. An adjective names a quality without showing what it looks like, so it leaves the judgment underspecified. Ask a model to make output 'sharper' and it has to guess which of a dozen things you mean.

An example shows what a criterion means in context. A strong client update can demonstrate how to state a decision and its deadline; a flawed one can show a plausible draft that omits the evidence behind its recommendation. Explain why each example succeeds or fails. The explanation helps the judge distinguish the principle from incidental details of one draft.

Use comparison sessions to collect examples that reveal a useful distinction. Keep an example for what it teaches the evaluator. A preferred version may demonstrate tone while still containing a factual error; a rejected version may include a paragraph to preserve. Label those limits, and add an example when it clarifies a criterion or exposes a failure your existing set does not cover.

Label each saved example by what it helps the evaluator understand. An accepted example can also demonstrate style, and a rejected example can sit near a boundary. These labels describe useful roles; the same output may serve more than one.

Example family	What it teaches the model
	What excellent output looks like in structure, tone, depth, and precision
Rejected examples	What plausible-looking output looks like when it fails on your actual criteria
Boundary examples	Cases where the call is genuinely hard, showing where your criteria have gray areas
Style examples	Tone, pacing, and format preferences that are hard to put into words

A tells a judge what to assess, and examples show how you apply it. Together they give the judge a clearer target. Keep some reviewed examples separate from the examples in its prompt. Those test whether the judge can apply the standard to work it was not shown while you designed it.

An scales your taste so more output gets checked against it

Use the criteria and reviewed examples to ask a model to evaluate a candidate. This is the pattern: an AI evaluator assesses particular qualities and cites evidence for its verdict. The judge may be a separate run of the same model or a different model. Independence of evidence and evaluation quality matter more than the label.

Give the judge the task, its criteria, and the evidence needed to assess them. For a client update, include the candidate draft, current project notes, and reviewed examples with explanations. State which requirements must be met and which qualities can trade off. Ask for a verdict on each criterion with supporting passages, an overall decision, and any missing evidence. A confidence label can flag uncertainty, but the cited evidence and tests of the judge determine how much weight to give its verdict.

A judge prompt built for the weekly client update

Task: You are evaluating a weekly client update email from a senior consultant to a non-technical client.

Priorities for this example: Factual support comes first: use the provided project sources and flag missing or conflicting information. Then assess whether the update makes decisions and material risks clear, gives supported deadlines, stays within the requested length, and ends with usable next steps. Do not invent a decision, risk, date, or number to satisfy the format.

Inputs: Supply the candidate update, current project sources, the , and reviewed examples with explanations of their strengths and failures. If the rubric lacks score descriptions, define them before using numerical scores. Ask for missing inputs before judging factual support.

Instructions to the judge: Use the 's 1-5 descriptions to assess each priority. Cite the candidate passage and, for factual claims, the relevant project source. Mark a criterion as unassessable when required evidence is missing. Give an overall verdict of pass, revise, or reject, with a reason; unsupported consequential claims require revision even when style scores are high. Prioritize revisions by their effect on the reader. State your confidence and explain what limits it.

Judge performance needs its own check. Try the judge on outputs whose strengths and failures have already been reviewed, including cases it has not seen. Give it the source evidence when a criterion concerns factual support. A word-count script can check length; a model may assess clarity or audience fit; a specialist may resolve a consequential domain question. Choose the evaluator by the property it can assess reliably.

A disagreement is evidence to investigate before revising the judge

A disagreement between you and a judge is a useful lead. It may reveal a missing criterion, unclear evidence, an unreliable evaluator, or a weakness in your own assessment. Investigate what supports each verdict before deciding which component to change. Agreement can also be misleading if both reviewers missed the same problem.

The cases below suggest places to investigate. The same disagreement can have several causes, so use the goal, source evidence, and repeated judgments to decide whether the , examples, evaluator, or your own assessment needs attention. A category helps direct the investigation; it does not establish the repair.

Disagreement family	What it reveals and where to repair it
Judge likes something you reject	The may omit a relevant criterion, or the judge may have found evidence you overlooked. Compare both verdicts against the goal and sources before changing the rubric.
Judge rejects something you like	The examples may fail to explain why the output works, or the judge may have caught a genuine defect. Check the evidence and add a boundary example if it clarifies the standard.
Judge is inconsistent across runs	The criterion may be ambiguous, the evidence incomplete, or the judge unstable. Inspect repeated judgments and test the smallest plausible repair.
Judge rewards length or polish	Check whether extra detail serves the reader or hides weak support. Give the judge examples of useful detail and empty padding, then test whether the distinction improves its calls.

Ask another evaluator to compare the competing verdicts with the goal and sources. It may identify a requirement to clarify, an example to add, or an error in either assessment. Test a proposed repair against the disputed case and cases that already worked. If the evidence cannot settle the disagreement, record the uncertainty and what would resolve it; adding a rule to force agreement would hide the problem.

Save each lesson where the next relevant task can apply it

A lesson becomes part of the workflow when a tested change carries it forward. Ask the assistant to propose the smallest useful update to the prompt, sources, tools, examples, , or sequence. Keep a previous version and compare outcomes before promoting the change. Adding a rule after every run can accumulate contradictions as easily as useful knowledge.

Save a lesson at the scope where it applies. A tone preference confirmed for client updates belongs with that workflow's instructions; a fact about this week's deadline belongs with the current task. A recurring handoff failure may require changing the sequence or what one stage passes to the next. Use the table to choose the relevant component, then check that the next task can retrieve the change. Broader scope is useful only when the requirement applies more broadly.

Component	What to update there
Memory or instructions	A preference or constraint you discovered that should persist across future sessions
	A requirement to add, remove, or clarify in your description of the deliverable
Prompt	An instruction to change based on what worked or failed this run
Examples	Whether this output should become a gold, rejected, or boundary example in your library
Judge	A criterion to revise after a disagreement you investigated
Workflow	Whether the sequence, review checkpoints, or handoff points should change

Use review triggers that fit the workflow

A new failure, changed source, model update, or revised goal is a reason to inspect the workflow. A scheduled review can also catch problems that individual runs do not reveal. Choose the trigger from the work you rely on and the cost of a missed error. The Deming Institute's Plan-Do-Study-Act cycle emphasizes comparing results with a prediction and learning from the difference; it does not prescribe a universal review calendar. PDSA Cycle (opens in a new tab).

These are possible review triggers. Pick the ones that solve a problem in your workflow, and adjust any scheduled interval based on what the reviews find.

Cadence	What to do
After an informative success or failure	Save the final output, the judge score if you used one, your own assessment, and any supported lesson likely to help future runs
When failures or judge disagreements recur	Review the recurring failures and disagreements with their supporting cases. Test whether revised examples or criteria address the problem before adopting them.
When the model, inputs, or requirements change	Re-run relevant reviewed examples with the changed model, inputs, or requirements. Revise prompts or standards where the results justify it, preserving the previous version for comparison.
Before expanding the workflow's authority	Re-test model choice for key workflows. Re-run trust reviews for any agent permissions. Decide whether a workflow should be promoted, retired, or rebuilt.

A reminder only creates an opportunity to review; it does not run the evaluation. If you automate the review, specify its inputs, checks, and reporting rule. Ask it to report meaningful failures and proposed improvements, and verify that it can read the examples and current standards it needs.

Verify a on recurring work before extending it

Choose a recurring workflow with accessible results and corrections, then test the evaluation process before extending it. The prompt below helps organize examples, criteria, a judge, and a way to capture lessons. Add review triggers that fit how the workflow is used. The loop itself is another system whose benefits need to be checked.

Build a learning loop around recurring corrections

Claude compares reviewed examples, proposes a rubric and judge, and designs checks for their reliability. You test the setup on held-out cases and choose review triggers appropriate to the workflow.

Copy the promptUse the copy button above the box.
Paste it in Claude or CodexMake sure your active project is enabled.
Answer the questionsGive clear answers so it understands what you want made.
Test it to see if it worksTry the output and refine until it does what you need.

Problem: Recurring AI work needs corrections, but I have not established which changes would improve later runs. Intent: I want a tested evaluation and feedback routine for that workflow. Ask for the workflow, intended reader, known requirements, current prompt or instructions, and actual outputs I found satisfactory and disappointing. Request the source material needed to check their factual claims and ask what I liked or rejected. If I can only describe an output, mark conclusions requiring its text as unresolved. 1. Compare the supplied outputs against the known requirements, sources, and my comments. Explain the evidence for each proposed criterion. Ask me to clarify uncertain preferences. 2. Draft a rubric with descriptions of strong, partial, and weak performance. If using scores, define the scale. Separate mandatory requirements from qualities that may trade off. 3. Draft a judge prompt that requests the candidate, relevant sources, rubric, and explained examples. Ask for evidence for each judgment, an overall pass/revise/reject verdict, prioritized corrections, and uncertainty. Include an abstain result when the available evidence cannot support a verdict. 4. Reserve reviewed final cases before tuning begins, including known errors and valid cases. Keep them and their expected verdicts outside the design conversation, and keep variations of the same source case in the same group. Before testing, agree on tolerable false-accept and false-reject limits, when to abstain or escalate, and the evidence the intended use requires. For consequential unattended decisions, identify the statistical method and assumptions needed to justify the case count and variety; do not invent a sufficient sample size. Lock the judge prompt, rubric, examples, model and settings, tools, and test procedure, including any repeated trials. Run the final cases in a fresh context without their expected verdicts and without changing that procedure. Compare recorded verdicts with the reviewed labels. Report false accepts divided by faulty cases, false rejects divided by acceptable cases, abstentions, and decision coverage: the share of all cases receiving a verdict. Include case counts, distinguish distinct cases from repeated trials, and explain uncertainty. Limit proposed use to what those results support; retain review when the evidence does not justify unattended decisions. If a final result informs a revision, use those cases as development evidence and require fresh untouched cases for the next final check. If suitable cases or actual results are unavailable, return a test plan and state that the judge is unvalidated. 5. Propose changes to the existing prompt, specification, sources, or checks only where the comparison supports them. Preserve the previous version and compare candidate changes on failed and previously successful cases before recommending adoption. 6. Return a named routine containing the rubric, judge prompt, record of results, and review triggers suited to the work. Record unresolved disagreements as well as supported lessons. Recommend a reminder only if a scheduled review would help. Build the routine from the material I provide; do not generate new outputs for the underlying workflow or invent reviewed test cases. Show inferred preferences with their supporting evidence and ask me to confirm before saving them as personal standards. Return the proposed routine here without creating a reminder or automation.

References

3 sources

1
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena View source
Zheng, Chiang, Sheng et al. · 2023 · NeurIPS
In the evaluated conversation benchmarks, strong model judges showed useful agreement with human preferences. The study also documented position, verbosity, and self-enhancement biases. These are reasons to test a judge, rather than assume its score is an objective measure.
Treat these findings as historical evidence of possible judge failure modes. Check the current judge on your own reviewed examples and give it the sources needed to assess factual claims.
The study evaluated 2023 models and conversation benchmarks. It does not establish the accuracy or bias rate of a current model on your workflow.
2
Reflexion: Language Agents with Verbal Reinforcement Learning View source
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao · 2023 · NeurIPS
Reflexion evaluated agents that stored verbal reflections from feedback for later attempts, without updating model weights. The reported gains occurred on particular benchmark tasks and feedback arrangements.
For power users, this translates to a practical principle: your AI system can learn operationally by improving the artifacts it consults. When you update memory files, specs, examples, rubrics, and workspace instructions, the next run reads the improved versions and starts from them, without retraining the model.
The experiments covered controlled decision-making, reasoning, and programming tasks. They do not show that every self-written reflection is correct or that adding memory always improves a personal workflow.
3
Demystifying evals for AI agents View source
Anthropic · 2026 · January 9, 2026
The engineering guidance combines code-based, model-based, and human graders, checks actual outcomes, and uses repeated trials because agent behavior can vary.
Choose a check suited to each criterion and test both the workflow and its evaluator. Include examples beyond those used to design the judge.
This is practitioner guidance from deployed systems, not evidence that a particular personal workflow improves after every revision.