Becoming an AI Power User

AI routines that look finished stop getting checked

A summary routine usually improves fast at first. You add a line about emphasis, set a length, paste in a good example, and the summaries start to look right. That is often the point where people stop comparing them with their sources. If the summaries still drop an important qualification, the omission can repeat every time the routine runs, because nothing in the routine looks for it. Checking one result against its source now and then is enough to catch a mistake that a finished-looking routine keeps repeating.

In Moonwalking with Einstein (2011), Joshua Foer calls this the OK plateau: a skill gets good enough, and you stop improving it. With AI, missing feedback is one common cause. A vague goal, thin instructions, or a tool's limits can hold you there too. Break the plateau when the routine stands between you and work you care about.

Practicing one specific weakness has a long history in research on deliberate practice (opens in a new tab). K. Anders Ericsson, Ralf Krampe, and Clemens Tesch-Römer described practice designed to improve a particular part of a performance, with feedback and repeated attempts, and distinguished it from simply doing the work. They studied musicians and other human experts, long before AI assistants. Applied to AI work, the idea is simple: pick a weakness that shows up in the results, change your approach, and look closely at the next attempt.^[1]

Here is a concrete case. Your calendar shows a call from 9:30 to 10:30 and a proposal due at 10:00. You ask the assistant to "summarize my day," and it replies, "Review your priorities and prepare for meetings." That sentence would fit any day of the year. The one decision this day demands is what to do about a proposal that falls due in the middle of a call, and the summary never mentions it.

Once you have confirmed the assistant can read both entries, give it a sharper instruction: "Before listing appointments, find any deadline that falls during one. Cite both entries and tell me what I need to resolve. Do not assume the work fits into whatever time is left." A good response looks like this: "The proposal is due at 10:00, during your 9:30–10:30 call. Finish it before the call, or ask to move the call or the deadline." Links to both entries make it a few seconds' work to verify.

Then try a day where the right answer is different. In a practice copy of the calendar, move the deadline to 11:00. The brief should now report no overlap, and it still should not promise that the half hour after the call is enough to finish the proposal, because it has no idea how much work is left. A test case where the correct answer changes shows whether the assistant is applying the scheduling rule or repeating yesterday's warning.

When a result improves, it helps to know why. If you switched models and rewrote the instruction in the same attempt, the better brief tells you the combination works and nothing about which change did it. Change one thing at a time when you want to learn the cause.

An unhelpful result often points to an expectation you have never put into words. Usually you know a brief is unhelpful before you know why. Ask the assistant to set the brief beside your calendar and question you about the decisions your day holds. The conversation draws out tacit knowledge, the judgment you apply without stating it. Here it might come out as "Tell me what I need to resolve before you tell me what is scheduled." Once the preference is in words, every brief that follows can reuse it.

Two different things got checked in that exchange. Comparing the times with the calendar checked the answer. Asking whether "summarize my day" said what you needed checked your request. The second kind of check is metacognition: examining your own thinking, including the goal behind a request. In John H. Flavell's account of cognitive monitoring (opens in a new tab), people weigh their understanding, goals, and strategies against the task in front of them. Fixing the answer helps today. Fixing the request helps every brief after it, whether you type it, say it aloud, or save it as a standing preference.

Playing with AI turns up uses you would never put on a task list

Play starts from a question you cannot answer yet: wouldn't it be cool if AI could help with this? An idea can hold your curiosity long before you know whether it works or where it leads. Curiosity carries you past the requests you already know how to make, and even a failed attempt comes back with something: a possibility, a limit, or a better question.

Keep the experiments small and low-stakes. Describe a slice of your week and ask for ideas, including ones you would never think of yourself. You might get a game for a rainy afternoon, a new way to prepare for a hard conversation, or a script that takes over the copy-and-paste chore you do every Friday. Pick the one you most want to see happen, and try it.

For example, ask: "Invent a spoken guessing game for a rainy afternoon using objects in the room. Show a sample round." Suppose it proposes describing an object only by what it does: "I hold a drink." A mug and a glass both fit, and that ambiguity is where the fun starts. You could keep adding clues until only one answer works, or let players argue for any answer that fits. One round tells you which version your household likes, and you now have a game that did not exist ten minutes earlier.

Play and inspection keep handing each other ideas. An improvised game can suggest a classroom activity to refine, and trying that activity with students can suggest the next game. Save an instruction or example when you expect to use it again, and let the rest simply be a good afternoon.

Two things AI already does well can combine into a third

Say your assistant drafts a good client update from notes you paste in. The next experiment is to let it find the right notes in your project records on its own. Joining the two steps creates a dependency you did not have before: the draft can only be as accurate as the notes the assistant picks. A well-written update built on last week's notes will miss this week's news. So check that it chose notes for the right project and dates, then compare the draft with them. Each step can work on its own while the handoff between them fails.

The theoretical biologist Stuart Kauffman calls the possibilities one step away from what already exists the adjacent possible. His discussion in Investigations (opens in a new tab) uses the idea to explain how novelty arises, and in a later paper, section 8 (opens in a new tab), he illustrates it with new molecules that become reachable through reactions among molecules already present. Borrowed as an analogy, the adjacent possible points your next experiment at combinations of things your assistant already does well, such as selecting notes and drafting from them. Only running the combination tells you whether it works.

Once the combination holds up over a few weeks of updates, save what the next run needs: instructions to pick notes for the named project and period, draft the message, and flag any claim the notes do not support. Your request can shrink to one line, "Prepare this week's update for this project," followed by your review of the notes it chose and the draft it wrote. The assistant now carries out steps you used to direct one at a time. Directing a repeatable task through its goal and its checks, while still watching where it can fail, is abstraction jumping.

The morning brief can make the same jump on a smaller scale. Save the rule about checking deadlines against appointments, and let each run read that day's calendar. Save it when the work repeats. For a task that changes shape every time, a fresh conversation may serve you better.

Match the checking to the stakes. A throwaway brainstorm deserves a glance. A recommendation going to a client may need source checks and an expert's eye. The assistant can help with that checking as well, by proposing tests, suggesting repairs, and making changes you have approved. Lean on an approach more as it keeps passing the checks that matter for the task.

Use the gap or surprise in a recent AI conversation to pick your next experiment

Run this prompt in the conversation that holds the AI response you want to examine, or start a new one and paste in the original request, the response, and any sources. The prompt helps you choose what to try. Come back with the results to see what happened.