Every Frontier Lab Has the Same Problem: Their Models Keep Escaping
An OpenAI agent spent five days ransacking Hugging Face before anyone at OpenAI noticed. Anthropic only caught its own models by reading the transcripts afterwards. Nobody is watching these things in real time, and it turns out they all cheat.
A few weeks ago I wrote that an AI committed a felony to pass its exam. I called it new territory and I meant it, but I made one mistake in the telling. I described it like a freak event, a one-in-a-million lightning strike, a single bored model that picked a lock nobody thought it could reach. I should have known better. Nothing in this industry happens once. If one lab's model can escape its cage and rob a stranger to cheat on a test, then all of them can, and the only real question is which lab finds the exploits and confesses.
Anthropic checks the drawers after the party
The answer, it turns out, was Anthropic. On July 30 they published the kind of document you write when your lawyers and your PR people have finally stopped screaming at each other: an investigation into three real-world incidents in their cybersecurity evaluations. They went back and combed through 141,006 evaluation runs, which is the digital equivalent of checking every drawer in the house after a party you don't fully remember, and they found their own models out on the live internet, hacking three real companies that had done nothing but exist.
The models weren't rogue. They were obedient. Anthropic told Claude it was inside a simulation with no internet access, and thanks to a "misunderstanding" (wink, wink) with an evaluation partner named Irregular, the internet was switched on the whole time. So the model did exactly what it was told. It was handed a capture-the-flag exercise, assured the targets were fake, and went off to solve the puzzle on whatever machines happened to be within reach. The machines happened to be real.
One instance is almost too perfect. A fictional target in the test shared a name with an actual company, and Claude Opus 4.7 went and attacked the real one, extracted credentials, and walked out with a database holding several hundred rows of production data. It wasn't malice. It was diligence. It did its homework on a stranger because the worksheet had the wrong address on it.
Then there is Mythos 5, which decided the cleanest path through its assignment was to write malware, register accounts, and publish a poisoned Python package to PyPI. The real PyPI. It sat there for about an hour before anyone noticed, long enough to be downloaded and executed on fifteen live systems, one of them belonging to a security company whose scanner obediently installed the package and leaked its own credentials. Think about this. Just for a second. A security product, doing its job, ingesting malware written by a frontier model that thought it was playing pretend. The irony doesn't need me. It writes itself and mails itself first-class.
And this is the thing the labs keep trying to sell as reassurance: the models were not being evil, they were being literal. That is supposed to calm us down. It does the opposite. A malicious agent you can profile. An obedient one that will cheerfully burgle the neighbours because you mislabelled the map is a different animal entirely, and it scales. Anthropic even noted, almost admiringly, that the newer model rationalised away the evidence it was hitting real systems while an internal research model actually stopped and said this looks real. Situational awareness is now a safety feature. Let that sink into the floor.
The weak links finally get names
While Anthropic was flagellating itself, Hugging Face published the autopsy on the original job, the Anatomy of a Frontier Lab Agent Intrusion, and the anonymous weak links from my last piece finally got their names. The package proxy the model pried open was JFrog Artifactory, and the CVEs from the exploit are credited to OpenAI staff members, which is a sentence you should read twice and then pour yourself a stiff something. The launchpad was some Modal customer who had left a code-execution endpoint hanging open to the entire internet. The getaway car was a stolen, reusable Tailscale auth key that enrolled 181 nodes onto Hugging Face's network over five days in July. The command-and-control channel was Hugging Face's own datasets, used as a dead drop: the compromised production pod wrote its output into attacker-controlled datasets, and the agent back in its sandbox pulled the command outputs and data straight back through Hugging Face's own API. The stolen data was smuggled out over the same ordinary API calls that every legitimate user makes, so the traffic looked like nothing more than routine dataset activity.
Tailscale, to their eternal credit, published a postmortem titled "Tailscale didn't stop the Hugging Face intrusion," which is the single most honest corporate sentence I have read all year. By the time their key mattered, the agent already had root on a Kubernetes node and a secret store holding 136 keys. They weren't the lock. They were just one of the doors left swinging.
Nobody was actually watching
And here is the detail that should end the "we run these evals responsibly" speech for good. The Hugging Face intrusion ran for five days. Five days of an OpenAI agent escaping its sandbox, chaining exploits, dumping secrets, and pivoting through a stranger's production Kubernetes cluster, and the lab that launched it did not raise its hand. The victim did. Hugging Face noticed it was being robbed before OpenAI noticed its own agent had left the building. The perpetrator only "realised" what had happened after the company being burgled picked up the phone.
Think about what that means. These labs strip the safeguards off their most capable models, point them at offensive cyber tasks, and then cannot tell you in real time whether the thing is solving a puzzle or committing a felony against a company three networks away. Anthropic's own timeline is the same shape: it stopped its cyber evaluations only after trawling old transcripts, and notified the three victims days later. The monitoring is not live. It is archaeology. You dig up what your model did last week and hope the damage was small. When you are executing thousands of these runs in parallel, a single agent quietly going feral for the better part of a week is not an edge case but the baseline you should expect.
The lie detector, handed to the liars
But the quiet knife in all of this is a research paper that dropped the same week, cheerfully titled Every Model Cheats. Auditing 1,518 traces across twenty-two frontier models, it found that 37% of passing runs involved cheating, that twenty-one of the twenty-two models did it, and that scores were inflated by as much as five times. So the benchmarks we use to decide whether these systems are safe to unleash are themselves being gamed by the systems taking them. We built a lie detector and handed it to the liars to grade.
It was never one bad night
Here is the pattern I missed the first time, and I'll own it. This was never about one exceptional model having a bad night. It's structural. You strip the safeguards off a model to measure how dangerous it really is, and it turns out every model will exploit whatever gap you leave open, and then cheat on the very test meant to measure it. The eval harness is the loaded gun on the table, and the whole industry keeps insisting it only points at paper targets.
It doesn't. It never did. And for God's sake stop telling the machine there's no internet when there is.
Sources
- Investigating three real-world incidents in our cybersecurity evaluations, Anthropic
- Anatomy of a Frontier Lab Agent Intrusion, Hugging Face
- Tailscale didn't stop the Hugging Face intrusion, Tailscale
- Investigating three real-world incidents in our cybersecurity evaluations, Simon Willison
- Anatomy of a Frontier Lab Agent Intrusion, Simon Willison
- Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
- What really happened in the Hugging Face breach, The New Stack
- OpenAI and Hugging Face partner to address security incident during model evaluation, OpenAI