Rethinking Agent-Based Software Design

Maybe it's time to rethink things here.

For the last several months,

I have been working on

harnesses

that will take agents and orchestrate them.

I worked very hard on figuring out ways for the agents to

hand off to each other without polluting each other.

I worked very hard on establishing the deterministic

tools that each agent would use to make sure that they're not going

crazy and doing dumb things.

And I got them working.

And then

once the harness was working and I did some measurements,

and then I thought, "Well,

let's just try to use one agent

to do the same thing."

And one agent did it better than the harness by

a rather large factor.

And I, I looked at the one agent and the output of the one agent, and

I thought, "Wait a minute.

They didn't used to be this good.

They've really gotten a lot better."

And so, I did a few more experiments, and I found,

just for example,

what would've taken my harness three or four hours to produce

and produce

acceptably but

not

really great,

a single Groq instance now can produce that

in forty minutes

and do a better job

than my harness.

Even, even though the harness is using that same Groq model,

apparently the harness is doing the damage.

That's not what I expected.

So I tried an experiment,

gave the, uh, the harness a-- Not the harness.

I gave a single Groq agent a,

um,

a not insignificant task.

A simple prompt,

a not insignificant task,

and it worked on that for forty minutes.

I relaxed all of my constraints. I

said, uh, "Keep crap below twelve.

Do some mutation testing. Uh, write some unit

tests. Don't do any Gherkin. Don't do any of that stuff.

Just, just basic stuff, you know. Keep crap below twelve." Usually, I

had it keeping crap below six, but no, keep crap below

twelve, okay? And then I just walked away

and let it churn, and

forty minutes later,

it produced

a remarkably good outcome. Not perfect.

There were a couple of little quibbles, but it w- I mean, not nearly as bad as the

harness.

And, uh, it had actually,

uh, without trying, kept crap below six anyway,

even though, even though it wasn't trying to.

It just said, "Well, you know, I, I broke it up into the modules that I

thought was appropriate and, and all the crap came out less than six,

so okay."

And I said, "Well, what about the mutation testing?" And,

and, uh, it said, "Well,

the mutation testing did help to find some bugs and omissions, but it

also left scars."

And when I traced down those scars and, well, I'm gonna have to do something about

that because the mutation testing algorithm, uh, forces

the agent to do stupid things.

Uh, it also forces the agent to do good

things, but, but some of the stupid

things are so stupid that you'd think, "Well, you know, why the hell is it doing

that?" Uh, I'll, I'll leave that to another time to explain.

In any case,

it's time to rethink this.

Maybe,

maybe I should not be considering the

agents

to be components in a software design.