Simplifying Processes in Technology

Well, it's easy to say simplify and it's very difficult to do it.

You know, I have this very basic first principles algorithm that I run kind of as like a mantra,

which is to first question the requirements, make the requirements less dumb.

The requirements are always dumb to some degree, so you want to start off by reducing the number

of requirements.

And no matter how smart the person is who gave you those requirements, they're still

dumb to some degree.

You have to start there because otherwise you could get the perfect answer to the wrong

question.

So, so try to make the question the least wrong possible.

That's what questioning the requirements means.

And then the second thing is try to delete the, whatever the step is, the part or the

process step.

It sounds very obvious, but people often forget to try deleting it entirely.

And if you're not forced to put back at least 10% of what you delete, you're not deleting

enough.

And it's somewhat illogically, people often, most of the time, feel as though they've succeeded

if they've not been forced to put things back in.

But actually they haven't, because they've been overly conservative and have left things

in there that shouldn't be.

So, and only the third thing is try to optimize it or simplify it.

Again, it sounds, these all sound, I think, very, very obvious when I say them, but the

number of times I've made these mistakes is more than I care to remember.

That's why I have this mantra.

So, in fact, I'd say that the most common mistake of smart engineers is to optimize

a thing that should not exist.

Right.

So, like you say, you run through the algorithm and basically show up to a problem, show up

to the supercomputer cluster and see the process and ask, can this be deleted?

Yeah.

First try to delete it.

Yeah.

Yeah.

That's not easy to do.

No, and actually, what generally makes people uneasy is that you've got to delete at least

some of the things that you delete, you will put back in.

Yeah.

But going back to sort of where our limbic system can steer us wrong is that we tend

to remember with sometimes a jarring level of pain where we deleted something that we

subsequently needed.

Yeah.

And so, people remember that one time they forgot to put in this thing three years ago

and that caused them trouble.

And so, they overcorrect and then they put too much stuff in there and overcomplicate

things.

So, you actually have to say, look, we're deliberately going to delete more than we

should.

It's that we're putting at least one in 10 things we're going to add back in.

Mm-hmm.

And I've seen you suggest just that, that something should be deleted and you can kind

of see the pain.

Oh, yeah.

Absolutely.

Everybody feels a little bit of the pain.

Absolutely.

And I tell them in advance, like, yeah, there's some of the things that we delete we're going

to put back in and that people get a little shook by that.

But it makes sense because if you're so conservative as to never have to put anything back in,

you obviously have a lot of stuff that isn't needed.

So, you got to overcorrect.

This is, I would say, like a cortical override to Olympic instinct.

One of many that probably leads us astray.

Yeah.

And there's like a step four as well, which is any given thing can be sped up.

However fast you think it can be done.

Like whatever the speed is being done, it can be done faster.

But you shouldn't speed things up until you've tried to delete it and optimize it.

Otherwise you're speeding up something that shouldn't exist as absurd.

And then the fifth thing is to automate it.

And I've gone backwards so many times where I've automated something, sped it up.

Simplified it and then deleted it, and I got tired of doing that, so that's why I've got

this mantra that is a very effective five-step process, it works great.

When you've already automated, deleting must be real painful.

Yeah, it's great, it's like, wow, I really wasted a lot of effort there.

Yeah.

I mean, what you've done with the cluster in Memphis is incredible, just in a handful

of weeks.

Yeah, it's not working yet, so I don't want to pop the champagne corks.

In fact, I have a call in a few hours with the Memphis team, because we're having some

power fluctuation issues.

So, yeah, it's kind of a, when you do synchronized training, when you have all these computers

that are training, where the training is synchronized to the sort of millisecond level, it's like

having an orchestra, and then the orchestra can go loud to silent very quickly, sub-second

level, and then the electrical system kind of freaks out about that.

If you suddenly see giant shifts, 10, 20 megawatts, several times a second, this is not what electrical

systems are expecting to see.

So that's one of the main things you have to figure out, the cooling, the power, and

then on the software, as you go up the stack, how to do the distributed compute, all of

that.

Yeah, today's problem is dealing with extreme power jitter.

Power jitter.

Yeah.

It's a nice ring to that.

So that's, okay.

And you stayed up late into the night, as you often do, there.

Last week, yeah.

Last week, yeah.

Yeah, we finally got training going at, oddly enough, roughly 4.20 a.m. last Monday.

Total coincidence.

Yeah.

I mean, maybe it was 4.22 or something.

Yeah, yeah.

Yeah.

It's that universe again with the jokes.

Yeah, exactly.

I just love it.

I mean, I wonder if you could speak to the fact that you, one of the things that you

did when I was there is you went through all the steps of what everybody's doing, just

to get a sense that you yourself understand it, and everybody understands it so they can

understand when something is dumb, or something is inefficient, or that kind of stuff.

Yeah.

Can you speak to that?

Yeah.

So I try to do, whatever the people at the front lines are doing, I try to do it at least

a few times myself.

So connecting fiber optic cables, diagnosing a faulty connection, that tends to be the

limiting factor for large training clusters, is the cabling.

So many cables.

Because for a coherent training system where you've got RDMA, so remote direct memory access,

the whole thing is like one giant brain.

So you've got any to any connection.

So it's the, any GPU can talk to any GPU out of 100,000.

And that is a crazy cable layout.

It looks pretty cool.

Yeah.

It's like, it's like the human brain, but like at a scale that humans can visibly see.

It is a brain.

Yeah.

I mean, the human brain also has a massive amount of the brain tissue is the cables.

Yeah.

So they get the gray matter, which is the compute, and then the white matter, which

is cables.

A big percentage of your brain is just cables.

That's what it felt like walking around in the supercomputer center is like, we're walking

around inside a brain that will one day build a super intelligent, super, super intelligence

system.