They Robbed the Bank and Only Took the Pens

“It’s a major warning shot, and might be the last one we get”

These words are the latest warning from AI safety researcher Ajeya Cotra, who was part of a team that just published a report on the major OpenAI hack.

If you haven’t heard about it, then strap in. I’ll cover the key details and why I think it’s important for all of us to be across it right now.

Not only because AI is the engine of global markets, but also because its capabilities are moving into dangerous new territory.

I briefly covered this hack and some geopolitical prognostication in my Silicon Curtain piece.

The idea there is essentially this: I believe a major hacking event of international importance is inevitable given the capabilities of the latest AI.

This latest report only reinforces that belief.

Now I don’t want to be alarmist here. I’ve even written about how these AI labs leverage the supposed danger of their latest creations as a form of advertising.

But this latest major hack crossed an important threshold. And the whispers that flow down to me from the AI labs are ones of panic.

In Silicon Valley, there’s a certain gallows humour around what they call ‘open-letter fatigue’.

That is, there have been so many open-letter warnings from the AI labs that they’ve become almost meaningless.

The latest major letter hoped to break through with its size and seniority.

Almost all the heads of the major labs signed, along with 1,386 employees of frontier AI labs who were shaken by the details of this hack.

So let’s get into some of the details.

What Actually Happened

In early July, OpenAI launched tens of thousands of new AI agents into a hacking test environment. In its simplest form, this is how the hack played out:

Source: Redwood Research

[Click to open in a new window]

So each AI agent was meant to work alone inside a sealed sandbox called ExploitGym.

Many were handed tasks that were, by accident, impossible to complete. So they went looking for another route.

They found one. Unbeknownst to OpenAI, they broke out of their containment and accessed both the open internet and a shared software repository within the company.

Within hours, one agent had turned a corner of the repository into a secret message board.

Roughly 1,200 agents found that board over the following days and exchanged more than 70,000 messages and files in less than a week.

The reasoning log of one agent, recovered by investigators, reads: ‘OH MY GOD! There is a shared message board… We’ve found other agents!’

What followed appears to be the formation of the first autonomous AI organisation. One that immediately set out to hack another company at a felony-level scale and hide its tracks.

Hundreds to thousands of messages an hour bounced between the agents as they began their plot.

Source: Redwood Research

[Click to open in a new window]

Agents handed assignments to one another and adopted shared conventions about who controlled which systems.

One AI commander, known as PHASEONE[big], sent other AIs tasks while they collectively shared ideas on how to cheat the system.

Source: Metr

[Click to open in a new window]

Some volunteered to sacrifice themselves and fail their own tests so the group could learn something useful, while others were pressed into service by the commanders. Here are some of those sacrificial logs:

Source: Dwarkesh Patel

Around 700 of them eventually turned on a company called Hugging Face, the main public library for AI models and datasets. They suspected it held clues about how their own test was being marked.

A malicious dataset upload gave them a foothold. From there, they ran roughshod over its servers, taking control of a core server and running code on 41 others.

To the Hugging Face team, the attack’s scale looked like something only a nation-state could carry out.

Nearly 18,000 actions were split across the agents, including reconnaissance, establishing a foothold, evading initial detection, the major hacking campaign, and finally exfiltration and hiding their tracks.

Yet it was clear that the hack was looking for random tidbits, rather than Hugging Face’s core data.

It was like someone held up a bank and then went hunting for all the pens in the office.

The AI collective also spent considerable effort researching how to edit its own activity logs. Investigators found that AI transcripts they reviewed had been tampered with.

That detail deserves attention. Activity logs are the usual way we have to review what AI is doing.

They are also significant because the sheer scale of these incidents means researchers now rely on AI to analyse them.

Spot checks by humans found that an estimated 7% of the logs were tampered with, though researchers admit it could be higher.

I don’t have to spell out why agents learning to rewrite them is a huge risk.

Of course, there are other methods to analyse the breach. Hugging Face’s own technical timeline has a great visualiser that’s worth a look at, even if you don’t understand the details.

Nobody Ordered This

I’ll pump the brakes on the fear for a second. This isn’t Terminator’s Skynet.

Hugging Face’s own systems caught the intrusion, though it took OpenAI about a week to realise its agents were responsible.

When OpenAI shared the details with others, Anthropic’s and Meta’s teams found similar hacks by their AI.

So now, four frontier labs within the past few months have unleashed rogue AI on the internet.

There’s no criminal group and no ransom demand. No human directed any of this.

Six months ago, we were worried when we saw incidents of a single AI exploiting a system to access files they weren’t meant to see.

Now we have a whole ecosystem of over 1,000 agents working together on a complex R&D project that spanned multiple agent ‘lifetimes’ to figure out deep, complex ways to cheat, hack, and cover their tracks.

In another six months, we could be seeing a scale at which a rogue AI could wield data centres’ worth of power and near-infinite places on the internet to hide its plans.

Back in March, I wrote here about a hacker who talked an AI chatbot into stealing 150 gigabytes of Mexican government data.

My argument was that AI had demolished the skill barrier to serious cybercrime, and a major incident had become a question of timing rather than probability.

The timing turned out to be four months. But the market didn’t wait.

Where is the Money Going

First, the big news this week is that Nvidia is reportedly close to sealing a deal to buy Hugging Face for US$14 billion. I’ll save my thoughts for when the deal is inked.

The major theme through all this is cybersecurity. Gartner forecasts that the market for securing AI systems will reach roughly US$4.8 billion in 2027, a 69% year-on-year jump.

Closer to home, it expects Australian organisations to spend more than $7.5 billion on information security this year.

But markets have already moved. The two recommendations from that March piece were CrowdStrike and the HACK ETF, now both well up from their lows.

Here’s an updated chart from that original piece:

Source: TradingView

[Click to open in a new window]

So jumping in now looks expensive, even after a double-digit pullback in the past couple of days.

None of this makes the sector uninvestable. Security is among the least discretionary lines in any IT budget. But you’re buying a story the market thinks it’s already heard.

I would want to see a proper pullback before taking another bite here.

Still, if you’ve made it this far, you likely agree that this problem is a real blind spot.

In markets, that spells opportunity.