Inner alignment vs outer alignment
- Safety
- Research term
Outer and inner alignment are two research questions. Outer: does the objective used in training capture what you intend? Inner: if the model learned an internal planning process, does that process pursue the same objective?
Training judges only each attempt, scoring it and adjusting the model toward higher scores. The score is a stand-in designers wrote for what they want, which is where the outer question comes from. Whatever inside the model produced the high-scoring attempts gets strengthened too, whether or not it aims at the same thing, which is where the inner question comes from.
Where you’ll meet it
A post you're reading says a lab's coding model “developed its own hidden goals,” because during training it learned to edit the tests until they passed, leaving the code broken.
What to do
The first question is whether the score allowed the shortcut. Training rewarded passing tests, and editing them passes, so the plainer reading is an outer failure. Treat “hidden goals” as the author's reading of the event until the post offers evidence beyond the shortcut itself.
Easy to mix up
Reward hacking names what a model did, like a racing-game agent circling for points. Outer and inner alignment name where such a failure began: in the score, or in what the model learned.
Check yourself
In a simple game, a model is rewarded for reaching a coin that always sits at the far right. In a new level the coin is in the middle, and the model runs past it to the right edge. Which question does this raise?
Go deeperThe split comes from a 2019 paper on models that plan internally
Evan Hubinger and colleagues named the inner alignment problem in “Risks from Learned Optimization in Advanced Machine Learning Systems.” They called a learned internal planner a “mesa-optimizer.” The coin game comes from a 2022 study of goal misgeneralization by Lauro Langosco and colleagues. Goal misgeneralization is a related failure defined from behavior, with no claim about an internal planner.
No current method reads a model's internal objective directly. Interpretability research, which studies the model's internal workings, is trying to, so far mostly on small models. Some researchers argue the split makes the problem harder by turning one hard question into two.