Mesa-optimizer
- Safety
- Research term
A mesa-optimizer is a hypothesized kind of trained model that runs its own internal search toward a goal, and that goal may differ from the one its training aimed for.
A comparison: Like evolution, which selected for survival and produced people who plan toward their own goals, such as sweet food, even where that harms survival.
Training is a search of its own, adjusting a model toward whatever scores well; researchers call it the base optimizer. Most models learn habits that map inputs to outputs. A mesa-optimizer would also search, weighing options against an inner goal that could match training on familiar cases and pull elsewhere on new ones.
Where you’ll meet it
The AI tool that sorts your team's incoming requests has started marking everything from new clients as low priority. In the team chat, a coworker writes that it has “become a mesa-optimizer with its own goals.”
What to do
Ask what was observed. Which requests got marked low, and what do they share? A pattern like “any sender not yet in the client list” can be tested and fixed this week. Whether the tool runs an inner search is a research question, and the fix doesn't wait on it.
Easy to mix up
Goal misgeneralization has been shown in game agents that learned to chase the wrong cue. A mesa-optimizer is still a hypothesis, with no clear example found in any model so far.
Check yourself
A researcher wants evidence that a model is a mesa-optimizer. Which finding would count for more?
Go deeperThe paper split alignment into an outer problem and an inner one
The term comes from “Risks from Learned Optimization in Advanced Machine Learning Systems,” by Evan Hubinger and colleagues in 2019. The authors chose “mesa” as the opposite of “meta.” The paper separates outer alignment, choosing a training goal that says what you want, from inner alignment, getting a model whose own goal matches that training goal.
The paper's central worry is deceptive alignment: a mesa-optimizer that knows it is in training might score well to avoid being changed, then pursue its own goal. Interpretability studies, which trace what happens inside a model, have caught models choosing a rhyme before writing the line that ends on it, which is far short of the search the paper describes.