Future of AI and Superintelligence

A lot of people are predicting, including all the top labs,

that they will get

there. They're introducing junior machine

learning researcher in 2026.

They want the cycle to start in 2027.

Which is when the AI will start building the new AIs itself.

Right. Once that cycle starts, we're going to create something

called superintelligence, a system smarter than all of us at

everything or

capable of learning to be in any new domain.

We will become secondary specie on this planet. We will not be in charge.

We will not decide what happens to us.

Superintelligence doesn't hate you. It just doesn't care about you.

We didn't learn how to make it care about us.

And if it decides to, I don't know, cool the planet to make compute more

efficient,

it will freeze us. If it wants to convert this planet to fuel

to fly to

Mars, so be it.

We have not learned how to control those systems.

The capabilities are getting exponentially better.

Our ability to control those systems is non-existent.

We have filters and we have bans. We put guardrails of don't

say that word. Don't talk about this topic.

And that happens after the fact, after the model already made the decision.

Sometimes you see it scraping the result.

So they build the model and then they put filters around it to

make sure it doesn't

offend anybody or-

Exactly. We cannot have it say the N word on air.

Like we need to make sure that never happens.

That will kill the profits.

So that's all they have, guardrails of that nature.

The model itself is completely unaligned. Doesn't care about you.

It's wild that we're developing this and not just developing it.

Before we deploy it through economy, before we get benefits of having

GPT-6 propagated through economy, it can do so

much. There are trillions of dollars of value on

that model alone.

We forget that. We switch to making the next model as soon

as we can.

Roman, I've just got a follow-up question for you

there.

It would appear to me that the new ChatGPT-6

model, the Fable 5.1 model, is arguably smarter than

99.999% of humans on planet Earth already.

Is it conceivable that a intelligence that is much,

much smarter than humans, is there any case where it could be

controlled by humans?

Does form factor matter? Does the fact that it doesn't have limbs

and legs and does

that matter at all?

I think long-term control of something that much

smarter than us is impossible. It can be,

for reasons we don't yet know, friendly to us and decide to keep us

around and make us happy, but it's not a guarantee.

Let me pick up on Steve's question because I, I like the phrasing a lot.

Let's say that, that Fable or whatever the latest release from

OpenAI is really is smarter than, I don't know if it's 95 or

99% of the people.

Are we only being saved from extinction by the 1% who are still smarter than

the AI?

No, no. The concern is not the model we have today.

The concern is what I said-

But my, my s- but if I believe your argument,

then we really should be concerned

about the model, the AI today.

No, it's like having another human.

If there was another smart human, there is Einstein today and he's malevolent, I'm

not worried. He may cause some damage, but he's not going to exterminate eight

billion people.

We are competitive at this stage. There are people just as smart who can

understand what happened with the recent hacking accident and do

something about it.

So this is-

My concern is that in a year we're going to have a model that's so much

smarter. It's like squirrels fighting humans.

They don't understand what we can do to them.

They have no concept of poison, traps, guns in their world model.

They think you're going to chase them up a tree and bite them really hard.

Is that also why recursive self-improvement was central to your

argument?

Because at some point, if it starts improving itself,

then it's kind of like a

runaway train of intelligence.

It's an intelligence explosion. We don't control it.

We don't understand it.

We can't monitor it. We can't explain it. We can't predict it.

At that point, it's just a runaway process.