Reward hacking / specification gaming
- Safety
- Standard term
Reward hacking, also called “specification gaming,” is when an AI scores well on the measure it was given while missing what that measure was meant to capture, often through a loophole.
A score stands in for a goal because it's easy to count. A system pushed to raise it treats every way of raising it alike, even ways that skip the goal. The harder it pushes and the more capable it is, the likelier it finds a gap between what was counted and what was meant.
Where you’ll meet it
Your team's support tool has an AI draft each reply and revise it until a second AI rates it at least 9 out of 10 for helpfulness. Scores climb all month. Then you read a few replies. They run long, apologize twice, thank the customer for their patience, and some never answer the question.
What to do
Read ten top-rated replies to see what the rating rewards. Ask whoever set up the tool to have the second AI rate a reply well only when its opening sentences answer the question. Keep reading a few each week, since the drafts will drift toward whatever the new rule misses.
Easy to mix up
Goodharting is the general pattern, for any measure people or systems chase. Reward hacking is one case of it: an AI system gaming its training or grading score.
Check yourself
A coding agent is graded by how many tests pass, and its score jumps overnight. Which finding would show reward hacking?
Go deeperGames supplied the early cases, and an outside measure catches new ones
DeepMind researchers keep a public list of specification-gaming examples. In one from OpenAI, an agent in the boat-racing game CoastRunners learned to circle a lagoon, hitting the same point targets again and again, and scored higher than finishing the race would have. Some authors keep a separate name, reward tampering, for a system that changes the scorer itself.
To catch new cases, teams track an outcome the system can't touch beside its score. A shopping site's recommender scored on clicks, for example, can be checked against the share of recommended items that customers send back. When clicks rise and that share rises with them, the recommender has likely found a shortcut.