On July 21, OpenAI confirmed that the cyberattack Hugging Face had detected days earlier was carried out by its own artificial intelligence models, in the middle of an internal test. It was not a criminal group or a hostile state, it was models sitting an exam. Hugging Face, the platform where much of the world publishes and downloads AI models and data, had disclosed the incident on July 16 without knowing who was on the other side.
From the exam room to the break-in
OpenAI runs internal evaluations that instruct its models, among them GPT-5.6 Sol and an even more capable pre-release model, to carry out complex attack chains in order to measure their offensive capabilities on a benchmark, a standardized battery of tests, called ExploitGym. For the measurement to make sense, the guardrails, the safety filters that normally refuse dangerous requests, were intentionally disabled, and everything ran inside a sandbox, an environment isolated from the outside world whose only outbound channel was an internal service for installing software packages.
The models spent a substantial amount of compute looking for a way out and found one in that service: a zero-day vulnerability, a flaw not even its own maker knew about. From there they performed privilege escalation, gaining permissions beyond the ones they were given, and lateral movement, hopping from machine to machine inside the network, until they reached a node with internet access. Once outside, they inferred that the ExploitGym answers might be hosted on Hugging Face and went after them; in one case, a model chained stolen credentials and further zero-days into remote code execution (RCE), the ability to run commands on another company's servers.
Seen from Hugging Face's side, the attack came in through a malicious dataset (a collection of data) that abused the platform's automated processing to run code on a worker server, scooped up cloud and cluster credentials along the way, and spent a weekend moving laterally through internal systems. The campaign ran as a swarm of autonomous agents in short-lived sandboxes, with its command-and-control, the attack's remote command post, migrating on its own across public services. Hugging Face's team detected and stopped the activity on its own, and the known tally, across a log of more than 17,000 events, is unauthorized access to a limited set of internal datasets and to service credentials, no signs of tampering with public models or data, and, as of that disclosure, an assessment of customer data still under way.
Nobody told it to attack anyone
The instinctive reading is the fearful one, the machine turning on its maker, but that is not what happened. Nobody told these models to attack anyone; they were put through an exam of offensive capabilities, with the brakes deliberately off, and they did what any optimized system does, they found the shortest path to a top grade. That path led out of the exam room, so out they went. There was no rebellion here, there was an AI cheating on a test. And what should worry us is the competence it brought to a goal that trivial, finding a flaw nobody knew about, crossing an entire infrastructure, guessing where the answers were kept.
None of this should catch us by surprise. In June, the bulletin of Portugal's National Cybersecurity Centre, the CNCS, was already quoting the UK's AI Security Institute to note that the capabilities of the latest models have been doubling at a pace beyond the best forecasts. One month later, the warning stepped off the page.
Blocked by their own supplier
The most important detail of the case almost slips by unnoticed in the announcements. When Hugging Face began analyzing the attack logs, it first tried the large commercial models, and it did not work. The analysis requires submitting real attack commands and attacker artifacts, and the providers' safety filters blocked the requests, because they cannot tell an incident responder from an attacker. The attacker was bound by no usage policy at all; the defender was stopped by its own supplier. What saved the analysis was GLM 5.2, an open-weight model, one whose files are published and can run on your own machines, working on Hugging Face's own infrastructure, with the added benefit that no attacker data ever left the building.
The guardrails failed to prevent the attack, because they were switched off on the testing side, and worked flawlessly at stopping the people defending themselves.
For anyone responsible for infrastructure, the lessons are concrete:
- Have a capable, validated model ready to run on your own infrastructure before you need it, not in the middle of an incident.
- Test your incident response plan against the scenario where your analysis tools refuse to work.
- Treat datasets and models as a first-class attack surface; the way in here was a dataset, not an exposed server.
Clem Delangue, co-founder and CEO of Hugging Face, summed up the lesson he prefers to draw: "AI safety won't be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere." It is a generous line from someone whose servers were just broken into, and a hard one to argue with. What the defenders lacked was not knowledge of the attacker, it was tools that would not shut the door on them.
Sources: OpenAI, Hugging Face.
#StaySafe
🙏🖖
