Becoming an AI Power User

When you scale your work from one task to hundreds, it is usually very important to design evaluation systems that catch errors in the process and self-heal them as you go. Past a certain volume you can no longer review every result yourself, and the mistakes that slip through are often the ones that leave a result sounding right. The design pattern behind these systems is much older than computing and has three parts: redundancy, so that no single copy or single checker is the only defense; quick feedback, so that a mistake gets caught close to where it was made; and evaluation inside the loop, so that checking happens during the work rather than after it.

Consider how long Vedic teachers have relied on this pattern, and what they trusted it to protect. For thousands of years, they have passed the Vedas, a body of Sanskrit sacred texts, from teacher to student by voice. The Rigveda alone contains more than 10,000 verses (opens in a new tab). Preserving it meant learning the words in order, with their precise pronunciation and tonal accents, then teaching another person to reproduce them. That person would teach the next, long after the first teacher could listen in and correct a slip. Whatever caught errors had to travel with the text.

Remembering the meaning of a passage would not suffice. A reciter could leave out a word or change a sound while still conveying its sense, and a listener who knew only the meaning would hear nothing wrong. If the change went unnoticed, a student could learn it and pass it on. Keeping thousands of verses intact across generations required ways to catch a slip even when the recitation still sounded fluent.

Students trained from childhood to recite passages in several forms: continuously, word by word, and in patterns that repeated words in different orders, sometimes backward before going forward again. Learning those extra patterns took more work, and it gave teachers and students several ways to hear whether a passage had changed.

In the overlapping-pair pattern, krama-pāṭha, each word is joined to the next: the first and second, the second and third, and so on. An English sentence shows how that catches an error. “Bring fresh water tomorrow” becomes “Bring fresh; fresh water; water tomorrow.” If “fresh” drops out, “Bring water tomorrow” still sounds sensible, and nothing in the sentence itself reveals the loss. The pairs do: “bring water” was never learned as a pair, and the learned pairs “bring fresh” and “fresh water” no longer fit anywhere in the sentence. The pairs catch the missing word only because they were learned from the teacher as a separate recitation. Pairs formed from the shortened sentence would agree with it and reveal nothing.

The comparison works because the learned pairs preserve information that the shortened sentence has lost. It helps expose the omission without guaranteeing that the learned versions are free of shared mistakes.

The learned pair “bring fresh” also supplies the missing word. The student can restore it and recite the passage again before teaching it to someone else. The extra repetition provides redundancy for detecting the omission and for recovering from it.^{[1] [2]}

An AI writing task allows variation that a recitation does not. “Could you send me your comments by Thursday?” and “Please send your comments by Thursday” make the same request in different voices, and either might suit you. “Send them whenever you have time” quietly drops the deadline. Checking AI writing means accepting a rewrite that keeps your meaning and catching one that changes it.

A reviewer agent checks an AI draft against your notes

Suppose you ask an assistant to turn rough notes into a friendly, direct email with the request up front. Your notes say: “Ask for comments on the report by Thursday. The sales figures are still estimates.” The assistant produces: “I hope this message finds you well. I'm reaching out to share the confirmed sales figures and would appreciate your feedback when you have a chance.” The grammar is fine. The email has also lost the deadline, promoted the estimates to confirmed figures, and buried the request under an opening you did not want.

Now give a second assistant the email, your notes, and your requirements. A reviewer agent is an AI assistant whose job is to evaluate work one requirement at a time, quoting the passages behind each judgment. This one should flag “confirmed,” set “when you have a chance” against Thursday, and point out that the request comes last. A verdict like “professional and well written” would miss all three problems.

The writing assistant revises: “Could you review the report by Thursday? The sales figures are still estimates.” The reviewer checks the new version against the same notes and requirements. Draft, review, revise, review again: that cycle is an evaluation loop. Your notes play the part of the learned pairs. They still hold the deadline and the qualification the draft lost, so they can put them back. If later drafts keep opening with formal introductions, fix the writing instructions and test them on another email. Feedback can improve both the current output and the instructions that produce the next one.

Your best past emails define the standard an AI reviewer applies

“Make it sound like me” is hard to check without examples of your writing. Give the reviewer a few messages you are happy with and say what you like about them. Perhaps they use ordinary words, put the request near the top, and sound friendly without a long warm-up. Ask the assistant to draft criteria from those observations, then read the criteria to make sure they describe you. One email you dashed off in a hurry should not become your permanent style.

Your standards describe the writing you want. A rubric restates them as questions a reviewer answers about each new draft. For the email above: Is the request easy to find? Is the tone friendly and direct? Did the deadline and the uncertainty about the figures survive the rewrite? A good rubric says what counts as meeting each standard and leaves room for more than one good answer. “Please send your comments by Thursday” should pass even though your example began with “Could you.”

Each rule also needs to know when it applies. “Include a deadline” makes sense for an email that asks someone to finish time-sensitive work. It makes none for “Thanks for explaining that; it cleared things up.” Write it instead as “When the message asks for work by a stated date, keep the date.” In the same way, a taste for direct requests does not mean a condolence note should open with an instruction. Give the reviewer the purpose of each message along with your style guide.

Run the review in a separate conversation that holds the task, the draft, your standards, the rubric, and the source notes. Then ask: “For each criterion, quote the relevant wording and explain your judgment. Say what evidence is missing. Suggest a change only where the draft misses a requirement.” A useful response reads like this: “The draft calls the sales figures confirmed, but the notes say they are estimates.” That finding is useful even if you later disagree with its opinion of the greeting. A single score would blur an unsupported claim and a matter of taste into one number.

Test an AI reviewer on drafts you have already graded

Before you rely on the reviewer, try it on writing you have already judged. The polished email that loses Thursday and calls estimates confirmed should fail. The accurate, direct revision should pass. Run each test in a fresh conversation with the reviewer instructions, the source notes, and one unlabeled draft, and keep your expected verdict and your own review out of it. Otherwise, the assistant might repeat an answer you supplied instead of applying the rubric.

Keep score: the mistakes it caught, the ones it missed, and the good passages it wrongly rejected. The tally shows which checks need work. A reviewer that reliably catches missing deadlines may still wave through an invented figure, so judge it one criterion at a time.

Add a good version with different wording, and a message with a different purpose, such as a thank-you that needs no request or deadline. These tests catch other reviewer mistakes: approving an unsupported claim, rejecting a valid paraphrase, applying an email rule where it does not belong. A reviewer has to recognize good work as well as find faults. One that demands changes to every draft will slowly sand off the warmth and personality your standards were meant to protect.

Suppose the reviewer rejects “Could you review the report by Thursday?” because its rubric says to avoid questions. The draft is fine, and the rule is wrong. Replace the blanket ban with what you meant: make the request clear and easy to answer. When the reviewer says a sentence does not sound like you, ask which of your writing samples supports that. Sometimes the evidence shows the reviewer caught a requirement you missed, and your own verdict is the one to change.

After fixing the rubric, test it on a new message as well as the sentence that prompted the fix. Does “Would you send me the revised slides before our meeting?” pass as a clear request? Rerun the earlier examples too. Dropping the bad question rule should leave the reviewer just as quick to catch a missing deadline or an invented fact.

Presentation can sway an AI judge, the usual name for an assistant doing this reviewing job. OpenAI's evaluation guidance (opens in a new tab) describes judge preferences related to answer order and length. When you ask a judge to compare two email drafts, run it again with their order swapped. It should not suddenly prefer the inaccurate version because that version came first. A longer, warmer greeting should not make an unsupported claim acceptable either.

Each kind of AI error needs its own check

A style review cannot tell you whether a report is accurate. Suppose your survey records show that 18 of 30 respondents preferred an online meeting, and an AI summary says, “Most customers prefer online meetings.” Ask a reviewer to trace that claim back to the survey. The source supports something narrower: “18 of the 30 people surveyed preferred an online meeting.” That count alone does not tell you what customers outside the sample prefer. The sentence needs the size and scope of its evidence, even though the broader claim sounds smoother.

If the assistant builds a slide deck from that report, check the exported slides as well as the outline. A chart labeled “80% prefer online meetings” conflicts with 18 out of 30, which is 60%. Even with the number fixed, the deck might tuck the sample size into text too small to read. Ask a reviewer to inspect the finished slides for accurate labels, legible text, and whether the deck does what you asked. A correct outline does not guarantee that the audience sees an accurate, readable slide.

A budget needs correct arithmetic and complete inputs. Suppose you have $500 for a weekend trip. Two hotel nights at $150 cost $300, and a $120 train ticket leaves $80 for food and everything else. If the assistant sets aside $120 for food, the plan totals $540. Ask it to recompute the total from the listed costs with a calculator or spreadsheet, then check what each quoted cost includes. The recalculation catches the $40 overrun. It cannot tell you whether the hotel quote left out a booking fee.

The same trip also has to fit you. If you asked to avoid early mornings and the itinerary starts with a 6 a.m. departure, the plan fails even if every price is right. Have the assistant compare the itinerary with each requirement you stated and list the conflicts. If it does not know a train's current departure time or a hotel's total price, ask it to look them up or mark them unresolved. It should not treat a plausible schedule as a confirmed one.

More reviewers help only when their checks can reveal something new. Two agents may both approve a report because both received the same outdated spreadsheet. Their agreement says nothing about whether the data are current. Ask one to compare the report with the figures and another to confirm that the figures came from the latest source. The learned pairs worked for the same reason: they came from the teacher, so they could disagree with the damaged sentence. Choose each extra check for the error it can expose.

Let a tested AI reviewer take the first pass

Your test results tell you how much of a recurring task to hand over. At first, read the writing assistant's revisions and the reviewer's explanations yourself. Once the reviewer consistently catches unsupported claims and accepts good writing on examples like the work you plan to delegate, let it do the first review and send failures back to the writing assistant. Ask it to bring you any source conflicts it cannot settle and any requirements it cannot judge, with the passages involved. What reaches you is the revised work plus the open questions.

The loop also needs a way to stop. If the writer and reviewer keep disagreeing about the same sentence, or a fix needs information neither of them has, have them send the disputed passage back to you with a note on what evidence is missing. Two assistants that agree after many rounds of rewriting have shown only that they stopped arguing. A successful correction should satisfy the failed criterion while preserving the parts that already passed.

For a weekly report, the assistant might write the draft, check its claims against the sources, recompute the totals, and review the wording against your style guide before you ever open it. Keep reading a sample of the reports that passed. If one still presents an estimate as a confirmed result, review the other claims that came from the same source and fix the check before you rely on the next report. The automated review is useful to the extent that it catches errors you would otherwise have to find yourself.

John D. Lee and Katrina A. See's work on trust in automation (opens in a new tab) distinguishes feeling confident in a system from relying on it. In your workflow, reliance comes as separate decisions: letting an assistant draft a report, letting it revise after review, letting it send the report to other people. Evidence that it writes in your style supports the first decision. It says nothing yet about whether its figures are accurate or whether the report is ready to send. Trust calibration means matching each responsibility you hand over to the work you have tested.

Closely related tasks can call for different judgments. In an experiment by Fabrizio Dell'Acqua and colleagues, consultants using GPT-4 did stronger work on tasks such as developing a new shoe concept and writing its launch material. On a separate task, they had to recommend which brand offered the best growth opportunity by combining financial data with company interviews, and consultants with AI assistance were less likely to choose correctly. The researchers call this unevenness a jagged frontier. The numbers belong to the older model they tested. The lesson for your own work is the unevenness: a persuasive slide deck is evidence about presentation, and the analysis behind its recommendation still needs its own check.^[3]

Keep the examples that exposed mistakes: the email that lost its deadline, the thank-you that needed no request, the report that overstated its evidence, the trip budget that ran $40 over. Run them again whenever you change a model, prompt, rubric, or source. Suppose a new style rule fixes the formal email and starts rejecting a warm personal thank-you. That is a regression: an improvement that breaks something that used to work. Add new examples as your work turns up new failures.

Build the checks into the recurring work itself: draft the report, verify its claims and totals, review it against your standards, revise what failed, and look over whatever is still unresolved. Each error caught in this week's report becomes a test case for next week's.

Run this prompt with a task you already repeat. Bring a few recent outputs, including at least one you know went wrong, and the sources or standards you judged it by.