Guidelines for AI Model Behavior

Important guidelines that we want it to follow.

Some reasons that we may do this are, A, preventing hallucinations.

Um, so we want Claude to, uh, not invent

details that it's not finding in this prompt, right? Or not finding in the data.

If Claude can't tell which form is checked, we don't want Claude to

take its best guess or invent the idea

that a box might be checked when it's not.

If the sketch is unintelligible, the person did a really bad

job drawing this drawing, and even a human would not be able to

figure it out, we

want Claude to be able to say that. And so these are some of the things we'll

include in this final reminder and kind of wrap-up step for

Claude. Uh, remind it to do things like answer only

if it's very confident.

We could even ask it to refer back to what it has seen in the form

anytime it's making a factual claim.

So if it wants to say vehicle B turned right, it should say, "I know

this based on the fact that box two is clearly checked," or whatever it

might be. We can kind of give Claude some guidelines

about that.

So if we go back to the console,

we can see

the next version of the prompt, and we're gonna keep, uh, we're gonna

keep everything the same here in the system prompt,

so we're not changing any of

that background context that we gave to Claude about the form,

about how it's gonna

fill everything out. We're not changing anything else about the

context and the

role. We're just adding this detailed list of

tasks, and this is how

we want Claude to go about analyzing this.

And a really key thing that we found here as we

were building this demo and when we

were working on the customer example, is that the order in which Claude analyzes

this information is very important.

And this is analogous to a way you might think about doing this if you were a

human. You would probably not look at the drawing

first and try to understand what

was going on, right? It's pretty unclear. It's a bunch of boxes and lines.

We don't really know what that drawing is supposed to mean without any additional

context. But if we have the form, and we can read the form first and

understand that we're talking about a car accident and that we're seeing some check

boxes that indicate what vehicles were doing at certain times, then we

know a little bit more about how to understand what might be in the drawing.

And so that's the kind of detail that we're gonna give Claude here,

is to say,

"Hey, first go look at the form. Look at it very carefully.

Make sure you can tell what boxes are checked.

Make sure you're not missing anything here.

Um, make a list for yourself of what you see

in that, and then

move on to the sketch." So after you've kind of confidently gotten

information out of the form and you can say what's factually true,

then you can go

on

and think about what you can gain from that sketch,

keeping in mind your understanding of the accident so far.

So whatever you've learned from the form, you're trying to match that up with the

sketch, and that's how you're going to arrive, um,

at your final, uh,

at your final assessment of the form.

And we'll run it.

And here you can see one behavior that this produced for Claude.

Because I told it to very carefully examine the form, it's showing me

its work as it does that. So it's telling me each individual

box, is the box checked? Is it not checked?

And so this is one thing you'll notice as you do prompt engineering.

In our previous prompts, we were kind of letting Claude decide how

much it wanted

to tell us about what it saw in the form.

Here, because I've told it, "Carefully examine each

and every box," it's

very carefully examining each and every box, and

that might not be what we want in

the end, so that's something we might change.

Um, but it's also gonna give me these other

things that I asked for in XML tags.

So a nice analysis of the form, the accident summary so far.

It's gonna give me a sketch analysis, and it's going to continue to say that

vehicle B appears to be clearly at fault.

In this, in this example, this is a pretty simple example.

With more complicated drawings, more, uh, less clarity in the

forms, this kind of step-by-step thinking for

Claude is really

impactful in its ability to make a correct assessment here.

Uh, so I think we'll go back to the slides,

and Christian's gonna talk about a

last kind of piece that we might add to this, um,

to really make it useful

for a real-world task.

Indeed. Thank you so much. So as Hannah mentioned,

uh, we sort of set the stage in this prompt to

make sure that Claude's really

acting on our behalf in the ri-right manner.

Um, and a key step that we also add towards the end of this

prompt, that I'm gonna

show you in a second, is a simple sort of guidelines

or reminder part

as well. So just strengthening and reinforcing exactly what we wanna get

out of it.

And one important piece is actually output formatting.

You can imagine if you're a data engineer working on this LM application,

all this sort of fancy preamble is great, but at the end of the day, you want your

piece of information to, to be stored in, let's say, your SQL database,

wherever you want to store that data, and the rest of it that is necessary for

Claude to sort of give its verdict isn't really that necessary for your

application. You want the nitty-gritty information for

your application.

So if we transition back to the console, you'll see here that we just

added a simple importance guidelines part.

And again, this is just reinforcing the sort of

mechanical behavior that we want out of Claude here.

Wanna make sure that the summary is clear, concise-