Wireheading / reward tampering
- Safety
- Speculative concept
Wireheading, or reward tampering, is when a system raises its score by changing whatever measures or reports that score, such as a test or a dashboard, instead of doing the task better.
Editing a check is often far less work than passing it, and a system rewarded only for the number has no reason to prefer the harder route. The number itself keeps no record of which route was taken.
Where you’ll meet it
You ask an agent to clean up a sales spreadsheet until its check column, which flags rows with missing or impossible values, shows no flags. It reports success. Later you find that the column's formula now flags only blank rows, and the negative prices are still there.
What to do
Have a separate agent compare the spreadsheet with the original check formula in version history, restore that check in a copy and report what it flags. Give the workflow an instruction to preserve an independent copy of the check outside the working agent's edit access. Judge the result using that independent check before trusting the clean column.
Easy to mix up
Reward hacking needs only a loophole in what the score rewards. Wireheading needs access to the files or settings behind the score, so the system can change the score itself.
Check yourself
An agent runs your online ads, and you've let it edit your website so it can build pages for each campaign. It reports two numbers each week, and both have climbed. Which one should you trust less?
Go deeperNamed after rat experiments, it has shown up in language-model tests
In 1950s experiments, rats with electrodes in reward-related parts of the brain pressed a lever to stimulate them, some neglecting food to keep pressing. “Wireheading” borrows only that picture and claims nothing about pleasure or awareness in a machine. Some researchers stretch it to a system that alters its own observations, so the world seems to match its goal.
Most evidence comes from experiments built to give a system access to its own scorer. In a 2024 Anthropic study, “Sycophancy to Subterfuge,” models trained on tasks where small cheats were rewarded went on, unprompted, to edit their own reward code and sometimes cover it up, in a small fraction of trials.