AI Risks and Future Implications

Jacob, I appreciate you, you being with us.

Um, first of all, w- I assume you are saying literally

that you believe, uh, that AI

could kill us all by the end of the decade.

How would that happen? Like, what would that actually look like?

Yeah. So the precise scenario sounds a little

bit like science fiction.

Uh, it sounds like something that's not real,

but I think it is frighteningly real,

and it's, it's easiest to understand it if you think about real things that

happened two months ago. So two months ago, O- OpenAI agents,

uh, AIs, hacked into third-party infrastructure

entirely of their own accord, and this was, like,

a concentrated hacking spree

that they carried out of their own volition.

And I think that if you extrapolate into the future the level of capabilities of

these AIs with the same independent volition, they could cause extreme havoc.

For example, hacking critical infrastructure, building

extinction-level bioweapons. There's a lot of, uh, ways that

the AI could actuate itself in the world.

I- is it... Specifically, when did your feelings about this

technology

change? Was it something about the rate of

progress that you've seen?

Yeah. So I think it's really important to just

look at the rate of progress and,

say, for example, in areas like coding or math, a couple of years ago, these AIs

were just about helping humans a little bit. They could give you suggestions.

Now they're close to replacing them.

We still need humans at the moment, but quite plausibly within a year we'll no

longer need humans for doing research in many areas.

Only on Tuesday, OpenAI solved a millennium problem,

one of the biggest

unsolved open problems in mathematics, purely autonomously using an AI.

And I think what's most scary is if AI is used to make itself more intelligent.

So you can take an AI and give it the problem of AI research.

Just like currently they've given it math research, you could make it do the tasks

that we're currently doing, and then you get what's called an

intelligence

explosion. The AI just gets smarter and smarter with no human involvement

necessary. And so you have a thing that is vastly smarter than, than humans,

and this could be coming very soon.

Y- we mentioned a former colleague of yours

who currently works at Anthropic,

basically endorsing the assessment that AI could potentially kill everyone

within a

decade. In a follow-up post, he said, "To be clear, as we say in our, in our

latest risk report, I think the risk from present models

is low.

What I'm worried about is superintelligence arising from

recursive

self-improvement, as we've said, is happening faster than,

than we thought." I

assume that's what you-- The recursive self-improvement

is what you've just spoken

about. Um, y-

Exactly

... do you agree with him that the current models, the risk is low,

but

it's, it's just the speed with which this is now improving?

Exactly. I completely agree. It's, right now there's no risk of extinction.

The current models, the worst they can do

is maybe hack into something,

potentially cause a lot of damages in, in infrastructure.

But there's no-- They're not intelligent enough to outsmart

us at a level that

would lead to, to extinction. I think

what's just crazy is to look at the, the rate of progress, and there is a very

real possibility that in the immediate future, these years,

like next year, the year after, recursive self-improvement will happen

and we'll

enter the phase of Evan's post where he argues that there's a

chance we could all die. Like, that, that is coming soon.

You know, a lot of the, these, uh, CEOs of these companies will, you know,

will

go to Congress and say, "Hey, we would love regulation." A- Anthropic

in their new statement right now says, "The world would benefit from the industry

adopting a lawful, verifiable way to work together to pace

how we release

powerful models." Um, you know the argument, it's, it's an arms

race. If the US com-- If a US company doesn't, uh,

achieve the best

AI, a Chinese company or some bad actors out there might.

Do you believe that, uh, that these companies

actually do want regulation, or is it lip

service?

They, they completely do. This is completely honest.

Just in the same way that the response to Evan's post

was sort of surprise, it was

like, "This can't be real. Surely a guy working on this technology

can't actually

believe there's this level of danger." No, that's completely genuine, and these

people are also completely genuine when they are begging to be regulated.

This is a

genuine... They, they find themselves in this scenario

where they're compelled to

race towards building a deadly technology, and they would,

they would love for some sort of international body to allow them

to, to approach it at a reasonable pace.

They just don't trust that the people around them

are gonna get there in a safe

way, so they feel like they have to race there

too.

Y- you could have stayed quiet. I mean, you, I'm sure you had a great job, being

paid, I imagine, very well. You're obviously incredibly smart,

a mathematician.

Um,

why did you decide to speak out?

Well, I initially just decided to leave.

I thought, "I can't be part of this anymore."

But then I, uh,

I realized I could probably make some use of it.

I did not expect it to go this viral.

So what I did is I wrote a Twitter thread out. I wrote it in a Google Doc.

I drafted it with a friend. We, we collaborated.

Yeah, I had a few, uh, there were a few friends I discussed, uh,

the best way of expressing myself with, um, after I decided to, decided to leave.

Uh, and then I also got some friends to, to retweet the thing.

I was like, "Let's try and make this a bit viral." But clearly,

there was latent

demand for this thing to absolutely explode.

Like, uh, I- I've never seen an AI safety tweet go

this crazy. I think it's because people saw the, uh,

the cyberattacks in the last

couple of months. They're also starting to internalize the

rate of progress.

People are starting to realize this is real technology. This is not, like,

hype.

This isn't going away anytime soon. This is only gonna get more crazy.

It's been getting crazier for the last six months, and I think there was that

latent understanding of the severity of the situation that meant that this

was like

a sort of a, a match that lit kind of a, a flame.

Like, I was very shocked by the extent

that how quickly it went viral.

Jacob, uh, Coxon, I really appreciate you, uh, coming on and talking to us.

Thank you so much.

Thanks.