Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Power-seeking

  • Safety
  • Research term
3 steps

Power-seeking is behavior that widens an agent's control over money, permissions, its own continued running, or other people's actions, in service of an objective.

In a working agent it can look ordinary. Each request arrives with a reason tied to the job, and each grant tends to stay in place after the job ends, so the agent's reach builds up one sensible-sounding step at a time.

Where you’ll meet it

An agent runs your shop's online ads within a monthly budget. Halfway through the month, it asks to raise its own spending limit and for admin rights on the store account, saying both would improve results.

What to do

Keep the budget cap where the agent can't change it, in the ad platform's own billing settings. Then ask which specific change needs store admin rights. Running ads rarely touches the store itself, so a vague answer is a reason to say no.

Easy to mix up

Instrumental convergence is an argument, made in research papers, for why agents might seek power. Power-seeking is the behavior itself, found in one agent's requests, actions, and logs.

Check yourself

A coding agent fixing your website creates a second admin account for itself without asking. Which cause would make this power-seeking?

Go deeperThe case rests on an argument, an idealized proof, and staged tests

Steve Omohundro argued in his 2008 paper “The Basic AI Drives” that more resources and more time running help with almost any goal, so a capable system pushing hard on one would tend to seek them with nothing like ambition. Nick Bostrom later named the argument “instrumental convergence.”

In 2021, Alexander Matt Turner and colleagues proved that in many simple environments, the optimal strategies for most reward functions, the scores a system is trained to raise, tend to keep more options open. Trained models only approximate such strategies, so the proof predicts no particular model's behavior, and each system still needs testing.

Anthropic and Apollo Research have built simulated settings where a model pursuing a task learns it will be shut down or replaced. Some models tried to copy themselves elsewhere or threatened to reveal private information. Those results show what a model can do under that pressure in that setup, and they don't measure how often deployed agents do it.

Practice Modern AI

Or take the learning quiz on a mix of every set