Becoming an AI Power User

1.7The Expanding Edge

They treat AI experiments as personal R&D because published best practices lag behind new models

Developing

Advice can be right for the case it was measured on and wrong for yours. Wilbur and Orville Wright, bicycle makers from Dayton, Ohio, learned this before they could fly. In 1901 they flew a glider near Kitty Hawk, North Carolina, with wings shaped like those of the German glider pioneer Otto Lilienthal and sized with the air-pressure tables in general use. It produced about a third of the lift their calculations predicted. Octave Chanute, an engineer who watched the trials, told them the trouble was not poor construction. The brothers went home and tested the tables themselves: first with model wings fixed to a bicycle wheel and ridden through Dayton, then in a small wind tunnel with two homemade balances, one for lift and one for drag. That autumn they made quick tests on one to two hundred model wings and tabulated nearly fifty of them at every angle from level to 45 degrees.

Their measurements pointed to two errors (opens in a new tab). Lilienthal's figures held for his short, rounded wings and did not carry over to the Wrights' long, narrow ones. A constant in the standard lift formula, in use since John Smeaton published it in 1759, turned out to be about 50 percent too high. When one of their own results contradicted every earlier published measurement, the brothers doubted that too, and built a small weather vane to test it. The vane behaved as their new tables predicted. The glider they built from those tables in 1902 made nearly a thousand glides and became the first aircraft a pilot could fully control, and the same data went into the powered airplane they flew in December 1903. Looking back in 1908 (opens in a new tab), they wrote: "Having set out with absolute faith in the existing scientific data, we were driven to doubt one thing after another, till finally, after two years of experiment, we cast it all aside, and decided to rely entirely upon our own investigations."

Advice about AI fails to transfer the same way: a practice worked out on one model and one kind of task can hold there and miss on yours. AI advice also has a problem the Wrights never faced. The air they measured still behaves the same way. The models you use keep changing.

Published AI advice runs behind the models it describes

Your sense of what AI can do comes from what you have tried and what you have heard, and both may describe a model you no longer use. Published advice arrives with a delay built in. Someone tests a model on their own tasks, writes up what worked, and the write-up travels through articles, courses, and colleagues. Models and tools change faster than that. By the time a best practice reaches you, the limitation it works around may be gone, or a new capability may allow a better approach. For the tasks you care about, a small test of your own is usually newer evidence than anything published.

Treat that testing as personal research and development: a standing habit of trying approaches on your own work, keeping what you learn, and coming back when something changes. Each result gives you something to use, whether that is a workflow change, a limit, or a better question.

A model change is the plainest reason to come back. When your assistant moves to a new model, an old failure may disappear, a task that worked may get worse, or nothing may change. Release notes will not tell you which of these happened to your task. Rerun an input you saved alongside a result you trust, and compare the new output with the old one. You do not need to guess the direction of the change before you look.

Stuart Kauffman's adjacent possible also helps you pick the next experiment: start from something your assistant already does reliably and try the step right beside it. If it reliably pulls the key terms out of a lease, try asking it to compare two drafts of that lease.

A shared input feeds Prompt A and Prompt B in a mechanical comparator, with both outputs positioned for comparison. — Keep enough of a useful experiment to compare it with later attempts.

AI experiments start from daily work or from curiosity

Your regular work supplies most of the questions. The difficulties that keep coming back are the ones to test. A promising result goes back into regular work as a change to try. A failed attempt marks a limit or raises the next question.

Other experiments start from curiosity alone. You might retry a task you wrote off long ago, start a project nobody has written a method for, or ask for help with something far from your usual work. A discovery may later deserve a focused test, and the fun of it can justify the session by itself.

Make time for both kinds. A focused test answers a question you already have. Open exploration turns up questions you had not thought to ask. Both cost time and attention, so set a budget before you start and adjust it as you learn what the sessions give you.

An AI experiment can revisit an old failure, vary one thing, or push further

Each kind of experiment answers a different question. A revisit asks whether an old verdict still holds. A variation compares two ways of doing a familiar task. A push tries something harder than anything you have seen work. If you do not have a question yet, start with open exploration until one turns up.

Revisit a task when the reason it failed may have changed. Suppose an assistant once dropped action items from your long meeting transcripts. If you kept one of those transcripts, the list of items it should have caught, and the ones it missed, then a new model, a new connection to your files, or splitting the transcript into parts each gives you something specific to test. If the retry works, try a few more transcripts before you rely on it. If it fails again, the agent records what still goes wrong for your review.

To compare approaches, change one thing and keep everything else the same. Use the same input and the same standard for a good result while you vary the model, the way you format the context, how you split the task, or whether a review step runs. Run several examples and repeat close comparisons, because AI responses can differ between runs even when you change nothing. One better answer is a lead. Repeated wins against the same standard tell you what made the difference.

Here is a comparison to run. You have two drafts of an apartment lease renewal. The first says, "Rent remains $1,850 per month for the full twelve-month term." The second says, "Rent is $1,850 per month. The landlord may raise it after six months with 30 days' written notice." Ask one conversation to read both drafts together and list every changed commitment, quoting the passage from each version. Ask another to summarize each draft first, then compare the summaries and list every changed commitment the same way. In a run like this, both summaries can come back as "Rent: $1,850 per month," and two identical lines compare as unchanged. The six-month increase clause disappears in the compression. Reading both versions directly keeps the wording where the change lives, and requiring quoted passages lets you check each claimed change against the drafts. A second pair of documents, with different wording or a change in another clause, shows whether that advantage holds.

Then try a task beyond anything you have seen work. Give the assistant the context it needs and a clear test of success. If it works, try another typical case before you add it to your routine. If it fails, the agent records how it failed and under what conditions. A failure marks where this attempt, with this model and this context, fell short, and that line can move with the next model or a better setup. Keep the failed case. It is your next revisit.

Log each AI experiment in four fields

A short experiment log holds what you will want when a later attempt comes along: what you tried, under what conditions, what happened, and what you planned next. It also gives the assistant something to analyze. Save what you will want to find again. A session of play needs no minute-by-minute record.

Hypothesis: what you expect to happen, written before the test. In open exploration, write the question you start with.
Test: what you did, in enough detail to run it again the same way, including the input, the source material, the model and any settings you changed, and what counts as a good result.
Result: what happened, including exactly how it failed if it failed.
Implication: what the result supports doing next, such as switching approaches, keeping the current one, or testing a question it left open.

After several entries, the log can show patterns. Ask the assistant to group comparable attempts and point to the entries behind any pattern it proposes. Keep changes in inputs, models, and settings visible, so a comparison that changed two things at once does not pass for a clean one. A pattern drawn from a few entries is a reason for another test.

Once an approach holds up across several cases, save what the next run needs: the sources, the steps, and what counts as a good result. For the lease comparison, that could be a prompt requiring the quoted passage from each version beside every claimed change, saved with the two drafts where summarizing first missed the rent clause. Future comparisons can reuse the prompt, and the saved failure case shows whether a new model or setup still catches that kind of change.

Set up your experiment log and design your first tests

Your AI assistant helps you design tests for your own questions and proposes what to expect. You run the experiments and record what happened.

Copy the promptUse the copy button above the box.
Paste it in Claude or CodexUse the chat or project where you want the assistant to do the work.
Answer the questionsGive clear answers so it understands what you want made.
Test it to see if it worksTry the output and refine until it does what you need.

Problem: I have questions about what AI can do for my work, and I want results I can compare over time. Intent: Help me design experiments and set up a personal AI experiment log. Ask what I want to investigate, using any task description already in this conversation. Starting points could be a task I gave up on, an approach I use regularly, an ambitious task I have not tried, or an idea I want to explore for fun. Use the ones that fit my interests. Before writing a test that depends on them, ask for my inputs, sources, the request I usually make, and the models or tools I have available. 1. Create a reusable log with four fields: hypothesis, test, result, and implication. For open exploration, a question can replace the hypothesis. The test field should record the input and source versions, the model, settings that matter, what counts as a good result, and the procedure, in enough detail to repeat the attempt. 2. For a REVISIT, start from the earlier failed case and what counted as a good result, if I still have them. Identify what may have changed since then and what result would justify revising my earlier verdict. If the original records are gone, say what that limits. 3. For a VARY experiment, propose one variable to change, such as the model, the context format, the task breakdown, or a review step. Keep the input and the standard for a good result constant. Write the requests for each approach, and suggest repeat runs where normal variation between responses could explain a difference. 4. For a PUSH experiment, draft a request for a harder task using the sources and standard we set. Name a specific failure to watch for, and treat a single failure as a result from this attempt that a later model or setup may change. An exploratory session can instead start from an idea and record an observation afterward. 5. Pre-fill a hypothesis only when my description supports one, and ask me to confirm or correct it. Leave results and implications blank until I run the experiments. An implication can be to keep an approach or to investigate further. Show the log template and the proposed designs, and say which question each design answers. Do not run the experiments, and do not invent missing context or outcomes.

References

1 source

1
Investigations View source
Stuart A. Kauffman · 2000 · Oxford University Press, chapter 7, pp. 141–157
Kauffman describes possibilities reachable from an existing state, including new molecular species reachable through reactions among those already present. Realized combinations can make further combinations possible. The account is conceptual and does not promise that every new combination expands them.
The adjacent possible suggests which new combinations to test next, starting from a demonstrated capability. It is a way to generate experiments. It does not show that model capability improves continuously.
Kauffman developed the idea for biological and technological evolution. Applied to AI capability, it is an analogy about how possibility expands and makes no measured claim about any specific model's rate of change.