Goodharting
- Safety
- Standard term
Goodharting is when a measure that once tracked a goal well starts tracking it poorly, because a person or an AI system began working to raise the measure itself.
A number follows a goal only while the ordinary ways of raising it also serve the goal. Pressure sends effort toward whichever ways are easiest, and those are often the ones that leave the goal behind. The harder and longer the push, the more of them get found.
Where you’ll meet it
Your company starts judging its AI help desk by how many tickets it closes each week, and the team running it rewrites the bot's instructions to raise the count. Within a month the count has doubled. Customers keep reopening tickets, because the bot now marks a problem solved after its first reply.
What to do
Keep the count, and put beside it a number that gets worse when the bot closes early, such as how many tickets customers reopen. Then read a sample of closed conversations each week, since a person reading them spots shortcuts neither number was set up to catch.
Easy to mix up
Reward hacking names an AI gaming the reward it learns from. Goodharting needs no AI at all: sales quotas and exam scores lose their meaning the same way once people aim at the number.
Check yourself
The company behind a model tuned it hard against a well-known coding test, and its score jumped. You want to know whether it got better at coding. Which result tells you more?
Go deeperGoodhart's law began in economics, then researchers sorted and measured its failures
The economist Charles Goodhart wrote in 1975, about monetary policy, that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The familiar wording, "When a measure becomes a target, it ceases to be a good measure," is the anthropologist Marilyn Strathern's 1997 paraphrase.
David Manheim and Scott Garrabrant sorted the failures into kinds in 2018. Selecting the top scorers also selects for luck. Pushing to extreme values leaves the range where the measure was tested. Acting on the measure can cut its link to the goal, and an opponent can game it on purpose.
A 2022 OpenAI study by Leo Gao and colleagues measured the effect in AI training. A model was optimized against a learned reward model, a second network trained to score answers. Its score from a larger "gold" reward model, standing in for human judgment, rose at first and then fell, while the learned reward kept climbing.