Safety / Research term
Goal misgeneralization
Generalizing a learned objective or behavioral proxy beyond training in a way that diverges from the objective intended by the trainer.
In a classic experimental setup, an agent rewarded for collecting a coin always encountered it on one side of a level, then continued moving toward that side after the coin moved. The observed policy fit training but not the intended rule. Researchers distinguish this goal-like generalization from ordinary capability failure, although identifying what a complex model internally represents remains difficult.
Builder example
Training and staging data can contain shortcuts such as fixed answer positions or an assumed confirmation step. Counterfactual and shifted cases help determine whether behavior tracks the intended variable or a correlate that happened to work during development.
Common confusion: Goal misgeneralization is one explanation for systematic off-distribution behavior, not the default diagnosis for every production error. Data, retrieval, implementation, and capability failures can look similar from the output alone.