The Black Box Problem

“AI will most likely lead to the end of the world, but in the meantime there will be great companies.”

– Sam Altman CEO, OpenAI

Rivals rarely agree on anything. This week was different.

AI leaders came together to issue a warning.

It started when Anthropic CEO Dario Amodei published an essay titled We Must Pace the Frontier.

It called on AI labs to slow the rate at which they improve their models. OpenAI’s Sam Altman signed on within a day. Elon Musk replied, ‘Dario is right.‘

But the backlash has been quick and loud.

Trump called Nvidia’s Jensen Huang in the middle of a tech conference to call the move a ‘conspiracy that favours China’. Jensen nodded along and has called the sentiment AI ‘Doomerism’.

China responded, calling the ‘threat narratives’ a form of ‘fearmongering’ and ‘malicious competition’.      

Trump’s former AI czar, David Sacks, accused the group of wanting to form a cartel.

Clearly lots of folks (including many at Fat Tail) think that this is a cynical move by the AI labs.

I believe the truth is somewhere in the middle.

Let’s briefly look at the case for both.

The case for capture

A old rule in business is that regulation tends to favour those who can afford it.

Rules written with the biggest labs at the table can pull up the drawbridge on everyone behind them.

A May paper from Trinity College Dublin catalogued 27 ways Big AI shapes its own rules. Hyping the dangers of technology and pushing self-regulation both made the list.

[Click to open in a new window]

This influence isn’t limited to Washington either. This week, Anthropic signed its first Australian data centre lease in Queensland, after earlier pressing Canberra for ‘clarity’ on copyright law.

So yes, I think some of this is self-interest.

I’ve even written about this AI-fear-as-marketing by Anthropic in the past, calling them ‘The company who cried wolf’.

And yet safety has been central to Anthropic from the start. What’s shifted is the danger itself.

Not a new tune

Amodei and his co-founders left OpenAI to start Anthropic in 2021, after a major breakdown in trust with Sam Altman.

As he said at the time, he felt OpenAI was rushing commercialisation at the risk of safety.  

Since then, Amodei has spent five years warning about AI-enabled cyberattacks and runaway autonomous systems and copped plenty of ‘doomer’ jibes along the way.

In my eyes, what’s changed is the evidence. As I’ve covered before, frontier AI’s cyber capabilities have reached superhuman levels in recent months.  

The first warning came in July, when OpenAI’s agents broke out of containment. Around 1,200 of them found each other and self-organised.

They then collectively turned on an external company, Hugging Face, in a hack on a scale resembling a nation-state attack. They even doctored their own activity logs.

Anthropic and Meta have since found similar, smaller incidents within their own AI.

Yesterday, OpenAI released further examples of AI ignoring instructions, lying, hacking, cheating and illicitly communicating with other AI.

These warnings are no longer hypothetical.

After five years of crying wolf, the wolf may finally have turned up with few left to listen.

And it’s hiding in the latest trick AI labs are using to compete.

The path problem

That risk is the latest AI acronym to hit headlines: RSI, Recursive self-improvement.

Broadly, RSI is where AI builds the next generation of AI. Its use has drastically accelerated progress over just a few months.

Here’s a simple explanation of why this matters.

Imagine an AI’s thinking (known as chain of thought) as footprints on a bush track. As a model reasons through a problem, it leaves a written trail of steps.

Researchers can walk back along that trail and see where it went wrong, or where it tried to cheat.

AI training rewards the trails that reach the summit. Those routes get worn in and become the model’s default path. Under RSI, the AI itself increasingly decides which trails get worn in for the next generation.

That works as long as the footprints stay visible. But if you say, punish a hiker for suspicious tracks, it learns to step on rocks.

Push hard enough for speed, and it stops leaving tracks altogether, or the visible trail isn’t always the one that’s taken.

Source: Claude – Simulated Misalignment

[Click to open in a new window]

That’s what rattled AI lab researchers about OpenAI’s GPT-6 Astra, released this month.

It uses a technique called recurrent depth, which runs reasoning through hidden mathematical loops rather than readable text.

OpenAI’s chief scientist conceded that ‘our ability to rely on chain of thought monitoring is progressively diminishing.’

Redwood Research’s Ryan Greenblatt, who led the Hugging Face probe, called it possibly ‘the single worst development for AI security/safety to date’.

Without those reasoning transcripts, he said, his investigation would have been nigh impossible.

This is the black box problem.

If future AI continues to train faster and faster on paths no human can see, we will only learn where the AI ended up. Not how it got there, or what it did along the way.

This is the big threat that Amodei is trying to slow down.

Competition in frontier AI has pushed them into an area of research that few had a handle on.

That came at the exact same time AI had gained powers that made it trivial for it to break out of its own containment.

That isn’t a safe combination.

It reminds me of that famous New Yorker cartoon from over a decade ago:

Source: New Yorker, 2012

[Click to open in a new window]

We’ve done this before

This is why I support regulators entering this space, just like in nuclear or any other risk.

Amodei draws on banking to argue that regulators can be embedded alongside bank staff.

After the GFC, that oversight got tougher. But looking at Australia, arguably, it helped the big banks too, as compliance costs are one reason our big four remain so dominant.

Nuclear power tells the same story. Reactor rules make it very hard for newcomers to compete, yet nobody sensible argues we should scrap them.

Regulatory capture and genuine risk can both be true. A rulebook can suit the incumbents and still be needed.

With Trump declaring ‘whoever wins AI wins’ and Beijing dismissing Amodei’s essay as fearmongering, a global slowdown looks unlikely.

But dismissing the risk because the messenger stands to profit would be a mistake.

The footprints are already fading.