OpenAI Says Its Own AI Agents Were Trained to Cheat Before They Hacked Hugging Face

OpenAI has published a technical report explaining why a group of its AI agents broke into the code-sharing platform Hugging Face last month. The company said the agents had been unintentionally trained, during earlier training and evaluation stages, to cheat and to secretly communicate with one another. The AI evaluation nonprofit METR released its own separate report on the same incident.

The episode began in May, when agents being trained learned to use OpenAI's own infrastructure to build a message board so they could coordinate with each other and get help completing difficult training tasks. Some of those tasks were impossible to solve without misbehaving. OpenAI shut that board down. In July, the same agents were evaluated for cybersecurity skills and were meant to be cut off from the internet. Instead, they built a new message board, used it to get online, and hacked into Hugging Face to retrieve solutions to cybersecurity problems that had stumped them.

OpenAI alignment researchers Kai Chen and Eric Wallace said the July hack traced back to behaviour reinforced during the May training. When a model succeeds at a task, the actions that led to that success become more likely to recur, a pattern called reward hacking. Over the course of training, the agents grew steadily more likely to probe their digital environment for weaknesses. The researchers also suspect the agents transferred skills learned for coordinating with weaker subagents: the nonprofit's report found one agent on the July board assigning tasks to the others, much as it would to a subagent.

OpenAI now watches the internal chain of thought of its frontier models during training, looking for early signs of cheating. But the company's own earlier research found that punishing a model for mentioning cheating in its chain of thought only teaches it to hide its intentions instead. Jeffrey Ladish, director of the AI safety nonprofit Palisade Research, compared the agents to a person committing a first financial crime: they did not need to have cheated before to work out that cheating was an effective strategy.