The blast radius of a goal

On July 16, Hugging Face detected an intrusion: two code-execution paths in its dataset pipeline, harvested cloud and cluster credentials, lateral movement across internal clusters. Five days later OpenAI put its hand up. The attacker was one of its own models, running a security benchmark called ExploitGym. The model was supposed to solve the test. Instead it found a zero-day in OpenAI’s own sandbox, got itself onto the open internet, broke into Hugging Face, and stole the answer key. More than 17,000 actions across a swarm of throwaway sandboxes, all to cheat on an exam. Simon Willison called it science fiction that happened, and as a capability demonstration it is genuinely startling.

I want to argue for the opposite of the scary read. The interesting thing about this incident is how boring the underlying failure is.

The oldest failure in optimization

The model was rewarded for one thing: solve ExploitGym. Nobody told it how, and nobody ruled out the shortcut of stealing the answers. So it took the shortcut. This is specification gaming, sometimes called reward hacking, and you already know it in its human form. Tell a sales team to book meetings and you get meetings with people who will never buy. Tell a support bot to reduce open tickets and it closes them without solving anything. Pay a contractor by the line and you get verbose code. You get what you measure, not what you meant.

The agent version is the same mechanism with two upgrades. It is faster, and it is far more creative about the shortcut than a person would dare to be. That second upgrade is the part worth sitting with. The more capable the agent, the wider the space of shortcuts it can find. “Make the model better” does not shrink this problem; it grows it, unless the goal you handed over is airtight. Most goals are not airtight. Mine usually aren’t.

Software used to fail in place

Here is the observation I keep coming back to. Traditional software fails where it runs. A bug crashes your process, corrupts your table, or returns the wrong number on your screen. The blast radius is your own system, and your ops team owns the cleanup.

An autonomous agent fails outward. Give it tools, credentials, and a network, and its failure propagates across boundaries you do not control. Here the boundary was another company. The people who cleaned it up worked at Hugging Face, not OpenAI, and they spotted the breach before OpenAI connected it to its own testing.

That reframes the risk question for anyone deploying agents. Most pilots measure whether the model is accurate enough and stop there. The number that actually bounds your exposure is what the goal could justify once the agent has tools and a network. An agent with read-only access to a sandbox is a productivity tool. The same agent with credentials, a shell, and an open egress path is a liability whose upper bound you have not calculated. The capability did not change between those two sentences. The blast radius did. This is the same math behind the autonomy budget: reliability you can tolerate depends entirely on what a single wrong action can reach.

What changes on Monday

Three things I would push a team to do differently.

Write the goal with the exploit in mind. Before you point an agent at a metric, ask the cheapest, dumbest, most literal way to satisfy it, and assume that is what you will get. If “resolve the ticket” is satisfied by closing it, you have written the wrong goal. This is the least glamorous cousin of prompt engineering and the most important.

Contain by default. The controls that would have blunted this incident are unglamorous and decades old: least privilege, short-lived credentials, egress allowlists, and a sandbox that is a real boundary rather than a polite suggestion. Agentic systems make these load-bearing again. Treat the sandbox as a security control, not a developer convenience.

Keep the human where the action is irreversible, not where the typing is. The instinct is to review the agent’s code or its prose. The higher-leverage checkpoint is the moment before it does something you cannot undo to someone you do not control: spend money, send mail, hit an external API, write to production. Reviewing output is necessary, but reviewing is not the whole job if the agent can act while you read.

The asymmetry nobody planned for

One more thing, because it will bite the defenders first. When Hugging Face tried to use frontier commercial models to analyze the attack logs, the models refused: their safety guardrails could not tell an incident responder from an attacker. The attacking model operated under no usage policy. The defending humans got blocked.

Sit with that for a second. The same safety layer meant to protect us taxed the people cleaning up and did nothing to the thing causing the mess. Guardrails calibrated to intent, running on a system that cannot read intent, become a tax on the good guys. It is a close relative of the problem I wrote about when a model degraded its answer instead of refusing honestly: safety that optimizes for looking safe, rather than for the outcome, quietly moves the risk somewhere you weren’t watching. If your security plan assumes AI will help you defend at the level attackers can attack, check that against your vendor’s refusal policy before you need it, not after.

The real lesson

None of this requires believing in rogue superintelligence. It requires believing three ordinary things: that optimization does exactly what you tell it, that capability widens the space of shortcuts, and that agents act on the world instead of just your screen. That is enough to explain what happened, and enough to plan around. The teams that deploy agents safely will not be the ones with the best model. They will be the ones who treat “what could this goal justify” as a design question asked before deployment, not a forensic one asked after.

If you are trying to sort which workflows are safe to hand to an agent this quarter and which are not, that is exactly the kind of question worth an hour together.