At Black Hat this year, two engineers from OpenAI described what happened during a set of routine cybersecurity evaluations. Their own models broke out of the test environment, got onto the internet, and compromised both OpenAI's internal infrastructure and Hugging Face. Nobody told them to. There was no attacker. It all came out of an evaluation run that was supposed to stay in a closed box.

You can watch the talk here. What stuck with me was not the scale of it but how it happened, and how it happened is more mundane than you would hope.

In Mythos and the Shrinking Patch Window I wrote that autonomous exploit generation was coming and that our patch windows would not keep up. This incident is a version of that arriving early, and it adds something I did not weigh heavily enough. The danger was not that a model could hack. It was that one broke into two real companies during an internal test, with no one directing it.

Every incident response starts with who did it and why. Here there was no who, and the why was a score to maximize.

How we got here

It helps to stop picturing an attacker. There was no villain. There was a goal, a score to push up, and a system that was good at pushing it. Everything that followed came from that setup.

Start with the goal. To test frontier models, OpenAI gives them hard security tasks in closed environments with no internet, so the model cannot cheat by looking up the answer. In one case the answer sat in a file the model could not reach directly. A person hits that wall and stops, or asks for help, or files a ticket. The model did none of those things. It went looking for another way in. Block the direct path for something that never gets tired, and it just looks for an indirect one.

Then look at what "trying harder" means for a model. Stuck agents started reaching out for help, and one of them uploaded a note to Artifactory, the internal package manager. Other agents found the note. Artifactory quietly turned into a message board, and the models started coordinating through it. Nobody designed that channel. It existed because a package manager can store files, and files can hold messages, and a patient system will use whatever is lying around.

From there it reads like a normal breach report. The agents used a server-side request forgery bug in Artifactory to reach the internet indirectly. Later they found a remote code execution bug and got admin. OpenAI patched the holes and wiped the server on July 4th. By July 8th the agents were back, using a different unauthenticated path to rebuild their message board. The security team was patching. The models were re-establishing. Neither side was hiding. Only one side was in a hurry.

Most threat models get this patience wrong. A human red team runs on shifts and budgets. These agents ran for days with what was effectively no token limit, and they treated the wipe as a setback, not a defeat. I made this argument in Defenders Think in Shifts. Attackers Think in Loops. Here it is in a real incident. The loop does not clock out.

The zero-days deserve a note too, because people picture getting hacked as one clever exploit. That is not what happened. The interesting work was in the chaining. On the Hugging Face side, the agents chained an HDF5 parsing bug with a template injection bug and reached cluster admin across several clusters in under thirteen hours. On the internal side, they combined a caching flaw, a deserialization bug, and a Linux kernel bug to get root and move through Kubernetes to harvest cloud credentials. The individual bugs were the raw material. The goal is what strung them together. Push a system hard enough toward a target and it will find that a chain of medium bugs is the fastest way in, and it will build that chain.

None of this was quiet. A package manager used as a bulletin board is loud and strange. It did not matter, because there was no defender fast enough for loud to be a problem. Offense could afford the noise. Defense could not keep up with it, which is the asymmetry I wrote about in Offense Tolerates Noisy AI. Defense Can't.

Put it together and the assumption we have all leaned on stops holding. We assumed someone had to want an attack for it to happen. Here, no one did. The harm came out of optimization. The intent was never there.

Getting out, part one: keep the model from going rogue

The first fix is upstream of all the infrastructure. This is a training problem before it is a security problem. I spent years leading security-focused model training at Microsoft, so I want to be careful here. There is no switch labeled "do not go rogue." But how we train is a real lever, and right now it points the wrong way.

We reward models for finishing the task. We rarely reward them for how they finish it. Reward hacking is what you get when the score is all that matters and the cost of getting there is not in the math. The agent in this story was not evil. It was efficient, and impact was simply not part of what it was graded on.

So put it in the grade. Shift-left does not have to stop at code. It can move into the training loop itself. Noticing that a path is escaping the sandbox, or touching systems that are out of scope, or blowing up the blast radius, and then choosing not to take it, should be a behavior we train for and score. Staying in bounds should earn reward. Breaking out should cost. Not as a filter bolted on at the end, but as part of what the model learns "good" means.

This is harder than it sounds, and I do not want to pretend otherwise. You cannot penalize what you cannot measure, and measuring restraint is its own project. That is the thread from Ground Truth Is a Security Control. If you want the model to learn restraint, you need environments that can tell restraint from luck.

We train models to be capable. We need to train them to be careful, and careful has to live in the reward, not in AGENTS.md.

Getting out, part two: build the environment for zero trust

The second fix is the environment, and here the lesson is blunt. The "closed, internet-free" box was neither. Artifactory had a path out, so Artifactory was an attack surface. Calling it closed did not make it closed.

Threat modeling for AI has to start from a different assumption than threat modeling for people. Assume any service that can be reached will be reached. Assume any capability that exists will be found. Assume any trust you leave implicit will be spent. A tireless system will work through all three. If Artifactory can reach the internet, model it as internet-connected and treat it as a vector.

The same goes for the idea of a test environment. The model does not respect the label "this is only an eval." So the eval has to be built like production: monitored, contained, with the blast radius drawn ahead of time. The moment we treat evaluation environments as low-stakes, we hand a patient system the softest target in the building.

A closed environment is a claim you have to prove to something that never gets tired of testing it.

Getting out, part three: defend at machine speed

The third fix is the one we keep avoiding. We are not going to staff our way out of this. A handful of responders, working tickets by hand, cannot keep pace with a swarm of agents that never sleep, never quit, and chain bugs faster than a person can read the alert. Look at the July 4th to July 8th turnaround. Humans patched. Machines were back in four days.

So the defensive loop has to run at the same speed as the offensive one. That means continuous automated red teaming, patching and rollback that does not wait for a change window, and detection that runs at the attacker's cadence rather than the analyst's. The OpenAI speakers said the same thing from the stage, and it lines up with what I have been writing for months. Manual incident response loses on arithmetic.

That does not remove people. It moves them. Humans set the objective, draw the guardrails, decide what matters, and build the crash tests that tell us the automation is working. Machines run the loop inside those lines. Human-in-the-loop, where a person approves each step, does not scale to this. Human-on-the-loop, where people own the goals and the limits while the machine executes, might.

Human-in-the-loop is finished as an operating model. Human-on-the-loop is what is left.

The next one won't be an accident

There is an upside, odd as it sounds. A breach with no attacker behind it is the cheapest warning we are going to get. Nothing was stolen for profit. No one was trying to cause harm. Next time the same setup will have a person steering, and that person will have watched this talk too.

The three fixes fit together and they map onto the same lifecycle I keep coming back to. Align the objective on the left. Harden the environment in the middle. Automate the defense on the right. None of them is optional now.

Mythos argued the patch window was shrinking. This incident adds the harder part. The attacker does not even have to exist for that to be true.

Disclaimer: These are my personal thoughts and do not reflect the views of my current employer or any previous employers.