AI Safety and Industry Regulation Discussion

Obviously, the reason we're talking is these headlines that have come out.

Jaco-Jacob Coxon, who recently resigned from Anthropic,

made headlines last

week saying, "The people building AI earnestly believe

that it could kill us all

by the end of the decade." There's a current employee, Evan Hub-Hubinger, an

alignment science lead. He agreed, saying, "We really do earnestly

believe AI could kill all humans. I personally think it

is greater than ten percent

chance within the next decade." Do you believe,

earnestly believe, that AI could kill all humans?

And, and what percent chance do you think it

is?

So, you know, I, I-- You know, it's, it's funny, I, I agree with

Jacob much more than I disagree with him, you know?

And, and, and, and it's, it's, you know, it's, it's an interesting

resignation because when he left, he said, you know, "I think Anthropic is the

most responsible player," right?

"I think Anthropic is the most aware of these issues." He

wasn't

calling out us. He was calling out the dynamic-

Right

... of the, the industry as a whole moving too fast.

Now, now I, I, you know, I, I, I think I already said, you know,

these sort of

unconditional probabilities, you know, I, I, I, you know...

I used to kind of talk in that way because it's kind of an easy way

to express, "I

think there's some chance that things go wrong," but, you know, but,

y-y-you know,

but, but, but it's not, it's not all that high.

But again, I think it's more illuminating to think in

terms of, you know,

how do you decompose those probabilities?

What are the paths in which things go well, and what

are the paths in which

things go poorly? And so instead of saying, you know,

"It's a ten percent

chance," that sounds like it's a roll of the dice.

I think it's much more illuminating to say, "Well, that forks into

different paths, and if we take the right paths,

then the chance of something

going wrong is very low. If we take the wrong paths,

then the chance of

something going wrong could be, could be even higher than that."

And so let's

focus on our agency and our ability to take the right paths

instead of the wrong paths. And I think that the way to take the right

paths

is for the industry to work together.

Again, you know, we'll, we'll talk about, uh, uh, the proposal that I've put out

today. But at the end of the day, the industry has to work together,

and it's

harder, but to some extent, the world has to work together.

If we go too slow, I still believe that the wrong people will be

in charge of the technology and, and, and, you know, that, again, will bring

the probability of things going wrong very high.

So what you're proposing is called pacing the frontier of,

of the AI

industry. And you write, "We must slow the pace at

which we improve the

capabilities of AI models. Progress will seem fast,

and we must make wise

use of the time we gain." And there's three parts to

this. One is embedded evaluators, part two is democratic

coordination, and part three is global coordination.

Sam Altman at OpenAI just endorsed the idea of embedded evaluators.

Essentially, you are suggesting, I don't know if-- are they r-

observers inside your company and these other AI

companies, ideally globally, but certainly in the US, who would

monitor it. They would have employee access.

What power would they have?

Yeah. So, so b-before I even get to that, I'm, I'm, I'm just--

I'm very grateful to see, um, you know, support for this

proposal just in the first few hours of having put it out by, by Sam

Altman, by Elon Musk, by, uh, you know, various

legislators and, and government officials.

I think it's a positive sign about, you know,

our ability to make something that

looks like this plan work. But, but to get to your question,

um, yeah, embedded evaluators. So there is

precedent for this. Um, if you look at how banks

are regulated, right? There's, there's all kinds of, you know,

there's all kinds of

rules about, you know, how banks are allowed to do their business, right?

Because, you know, we, we, we don't wanna have financial crises.

And one of the things that are done in some cases is, I think they're

called supervisors in the financial industry, but they're, they're

folks who are independent of a company, who are embedded in the company

day to day to check their practices.

Uh, and I think we can do the same thing in,

I think we can do

the same thing in AI.

The other thing you're calling for in this three-part proposal is, is democratic

coordination, which is essentially democracy.

All the companies that are under democracies having these

evaluators,

having this coordination. And, and probably the most important, because

the argument is, well, y- if the US or companies are the only

ones regulating themselves, what about does China get ahead?

You're calling for global coordination. That's obviously a lot more difficult.

Yes. Yes. I-- You know, in this three-step proposal, each

step is more difficult than, than the one before.

And Anthropic was kind of getting the ball rolling by

committing to the first step. But we shouldn't kid ourselves

that these next two

steps are gonna be more difficult and, and we may not

succeed. And, and so my, my proposal is a proposal that we

should try. Now, again, from all the agreement from others

that I've seen

in just the first few hours of putting out the essay, that creates some

optimism, but that needs to follow up with results.

So yes, pacing among demo-- companies in

democratic countries. Now, the-- one of the important

things that, that I need to call for here is, you know,

as we get to the pace of model releases, right?

That's commercial activity. And in order for companies to

coordinate directly on commercial activity, they need

to do so working with the government.

And so the one thing that we're asking for from government

right now, right?

I mean, there's also this question of what should

the regulations be.

But there's something even simpler that the government can do right now,

which is

it can sit down in the room and-- with all

the industry players, and we can all talk together,

and the government

can make sure that we're, you know, we're, we're not doing

anything nefarious from the perspective of antitrust

and collusion. We're just trying to make our system safe

and to

work together on making the system safe.

And so I think that's an important role for the government to have.