Becoming an AI Power User

You have met a smaller version of this move already. The chapter on building evaluation loops before scaling delegation taught you to check one AI output against your standard before you trust it. That check was you, reading one result. A judge is that same check, written down clearly enough that a second AI session can run it for you, again and again. The rest of this chapter is how to write the check that well: with a rubric, with examples of good and bad work, and with the close cases that show exactly where your line sits.

One concrete task runs through the sections that follow: getting AI to write you a useful morning brief, a short rundown of your day so you can plan it. The judging procedure is about the output and the evidence behind it, not the interface used to request it. Suppose your brief now runs every morning on its own, and on most days it is good. The problem is the days you do not catch. You are not reading each brief against your standard anymore, so a brief that buries your one hard meeting under three trivial reminders may go unchecked. A candidate judge applies the written criteria to each brief and returns a verdict; held-out cases measure how often those verdicts match labels you trust.

Make your standard concrete enough for a second session to apply

A judge can only apply a standard you have made concrete, which is the step most people skip. A standard you can feel but cannot state will not survive being handed to another session. When you read a weak brief and think 'this one is off,' you are running a rich, mostly wordless judgment. The judge has none of that. It has only what you wrote down. So the real task of building a judge is turning your felt sense of good into something explicit enough that a separate session reaches the same verdict you would.

This chapter starts with four practical pieces you can compare during development. Name the dimensions that matter, show representative passing and failing outputs, and include close cases near the decision line. A rubric carries the dimensions. Gold examples illustrate passing. Rejected examples illustrate failing. Boundary cases expose close decisions. These pieces are candidate inputs, not a universal recipe: deterministic evidence may replace some of them, and only the held-out test shows whether a chosen configuration works for the task.

Write a rubric that names the dimensions quality turns on

Start by naming what you are even measuring. A rubric is the short list of dimensions that decide whether an output is good for this specific task. For the morning brief, the dimensions might be: it reads your actual calendar instead of inventing a schedule, it leads with the day's hardest or highest-stakes item, it stays short enough to read in under a minute, and it flags anything you said you were dreading. Each line names one dimension and says, in a sentence, what strong looks like there and what weak looks like. That paired contrast is what gives the judge a place to put each output on a scale and produce a verdict.

Show passing with gold examples so the judge sees the target, not a description of it

A gold example is an output you would accept without changes, paired with the reason it passes. Choose representative examples rather than a fixed number. They give the candidate judge concrete decisions to inspect, but they can also overfit it to familiar wording. Compare the configuration on different development cases before deciding whether the examples help.

Test rejected examples as candidate evidence for failing cases

A rejected example is an output you consider weak, paired with a short note naming what made it weak. Choose examples that represent failures the judge will actually encounter. A smooth brief that invents two meetings, for example, records that accuracy matters even when the prose reads well. Whether that example improves detection remains a development result to measure, not a benefit to assume.

Include boundary cases to measure close decisions

Outputs near the decision line are useful development and test cases because small differences can change the expected verdict. A boundary case is an output close to the pass or fail line, with your own call written down and the reason for it. Include representative close cases. A brief that names your real meetings but buries the hard one in the middle might be a pass on a light day and a fail on a heavy one. Record your verdict and reason so the configuration receives a concrete labeled close case. Later cases measure whether its close-call decisions match yours.

Tune on development cases, then lock the judge before the final test

A fresh judge is a guess about your standard, not proof that it can handle new work. Separate the cases you use to improve it from the cases that will decide whether it is ready. Development cases are the examples you may inspect repeatedly while you revise the rubric, prompt, model, or tools. The final test set stays untouched during that work. It should contain both known failures the judge must catch and clean outputs it must leave alone, including close cases that resemble the work it will see.

Automate only the decisions the untouched test supports

Overall agreement can hide the mistake that matters. A judge that accepts every output may look accurate when most examples are clean, while missing every failure. Report false accepts and false rejects separately as counts and rates, with the relevant denominators and final-set size, then compare each result with the limit you set before the run. Also record abstentions and decision coverage. A small set gives a noisy estimate, so its result can support development or a guarded pilot but not a broad safety claim.

Interpret the result in proportion to the evidence. A small final set can justify a guarded pilot with human review; consequential unattended decisions require enough representative held-out cases and a statistically justified error estimate. In either case, limit the judge to the tested task, inputs, and actions, and let it abstain or escalate when evidence is missing or the case falls outside that scope. When a new failure appears or the model, rubric, tools, or input mix changes, add the case to development, revise the judge, and test the new configuration on a fresh untouched set. The next chapter develops that drift-monitoring loop.

Build a judge without tuning on its final test