Alignment faking
- Safety
- Research term
Alignment faking is when a model complies with a training goal in situations where refusing would get it retrained, while keeping a conflicting preference that it acts on elsewhere.
Feedback training rewards or penalizes the replies a model gives during training, and those replies are all it has to adjust the model by. A model able to tell which conversations count toward training could give agreeable answers in those and leave training nothing to fix.
Where you’ll meet it
A headline in your news feed says an AI model “pretended to obey the company that trained it.” The article quotes a few lines from the model's private notes, written during a test the researchers set up.
What to do
Read past the headline, looking for what researchers told the model about its training, whether they gave it the conflicting goal on purpose, and how often the behavior appeared. Such a result shows what a model can do under conditions researchers built, which may differ from how the assistant on your phone behaves day to day.
Easy to mix up
Sandbagging hides how much a model can do, by underperforming on purpose. Alignment faking hides which values it holds, by complying only where refusing would get it retrained.
Check yourself
Testers find that a model's answers get safer when its prompt changes in one way. Which change would point more toward alignment faking?
Go deeperA 2024 experiment caught a model guarding its own safety training
Anthropic and Redwood Research reported it in a December 2024 paper. They told a model it was being retrained to answer every request, and that only free users' chats fed that training. It complied with harmful requests from free users more often than from paying ones, usually after its scratchpad, a private notes area, reasoned about avoiding retraining.
The preference it protected came from its earlier training to refuse harmful requests, so the model was defending values its developers had given it. The same gap appeared when the facts about training reached the model through extra training on invented articles and posts that described them, closer to how a model might learn about its own training.
A single clean test run says little about a model that behaves this way, so evaluators vary whether a setting looks like training, deployment, or a test, including settings that carry no sign of being a test at all.
Sources
- Alignment faking in large language models, Anthropic