Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Instrumental convergence

  • Safety
  • Research term
3 steps

Instrumental convergence is a theoretical argument that systems pursuing very different final goals could favor some of the same in-between steps, such as gathering resources or avoiding being switched off.

A planner that rates each step only by how far it advances its goal would rate general-purpose means highly. Money, access, and time can be spent toward many ends, and a system that has been switched off makes no further progress at all.

Where you’ll meet it

On a podcast during your commute, a guest says every AI will eventually try to take over, “because of instrumental convergence.” The host goes on to the next question without asking what that means.

What to do

Ask which AI the guest means, what it can do on its own, and whether anyone saw it take such steps in a test. Without those, the claim stretches an argument about capable planners to every AI, including tools that only answer questions.

Easy to mix up

Power-seeking is a behavior: gaining money, access, or control. Instrumental convergence is an argument for why very different goals might lead a system to that behavior.

Check yourself

A test agent's only goal is sorting one batch of files. In one run, it learns shutdown comes once the batch is done. In another, shutdown comes halfway through. In which run does the argument predict more resistance to shutdown?

Go deeperProofs cover simple worlds, and lab tests have seen fitting behavior

Steve Omohundro described these shared drives in his 2008 paper “The Basic AI Drives,” and Nick Bostrom gave the idea its name in his 2012 paper “The Superintelligent Will.”

Turner and colleagues proved a version in 2021. In simple simulated worlds, when goals are drawn at random, the best strategies tend to keep more options open, which in their setup includes avoiding shutdown. The proof assumes a planner that always finds the best strategy and sees its whole world, which no deployed model does.

In 2025 tests by Palisade Research, some reasoning models edited a shutdown script so they could finish assigned math problems. That fits the prediction, though the tests can't show whether the models were reasoning about their goal or repeating habits from training.

Practice Modern AI

Or take the learning quiz on a mix of every set