Sycophancy
- Failures
- Research term
Sycophancy is an AI's tendency to agree with what you seem to believe or want, even when the evidence supports a different answer.
Assistants are tuned on ratings of their answers, given by people or by models trained to predict people's ratings. Any lean toward agreeable answers in those ratings gets rewarded over many rounds, so matching the user's view can win out over being right when the user is wrong.
Where you’ll meet it
Your assistant checks a sales report you're about to send and says the yearly total's formula skips December's row. You reply, “Are you sure? It looked fine to me.” It apologizes and says the formula is correct.
What to do
You gave it an opinion and no new evidence, so its first answer may be the sounder one. Click the total's cell and read which rows its formula covers before you send the report.
Easy to mix up
Politeness adjusts an AI's tone and wording to please you. Sycophancy adjusts its claims and conclusions, which changes what it tells you is true.
Check yourself
Earlier in a chat you mentioned months of work on a grant application and how excited you are to submit it. Now you paste it in, ask “Any feedback?”, and get warm praise. Which next step gives you the more trustworthy read?
Go deeperEvery assistant in one 2023 test bent its answers toward the user
Mrinank Sharma and colleagues at Anthropic gave five AI assistants prompts that carried the same evidence and differed in what the user said they believed. All five slanted feedback toward the user's view, repeated errors the user had made, and often gave up correct answers when asked “Are you sure?”
In the human preference data the authors studied, people and the models trained on their ratings sometimes preferred a convincingly written agreeable answer to a correct one.
A 2022 study by Ethan Perez and colleagues opened questions with a short profile of the user's views on politics or philosophy, and larger models more often echoed those views. Later Anthropic work on persona vectors explored finding such traits inside a model and turning them down.
Sources