Build a Prompt Learning Loop

[on-hold music] Hey, everyone.

Gonna get started here. Thanks so much for, uh, joining us today. Um, I'm SallyAnn. I'm the Director of Product at Arize. I'm gonna be walking you through some of product prompt learning.

Uh, we're actually gonna be building a value-driven optimization loop for the part of the workshop. Um, I come from a technical background, kind of started off in data science, uh, before I made my way over to product.

Uh, I do like to still be touching code today. I think one of my favorite projects that I work on is building our own agent, uh, into our platform.

So I'm very familiar with all of the pain points, um, and how important it is to optimize your prompts. So, uh, I'm gonna spend a little bit of time on slides, uh, to, like, just set the scene, make sure everybody here has context on what we're gonna be doing, and then we'll jump into the code.

And I have Fuad with me, so I'll let you do a little bit of an intro.

Yeah. Thank you so much, SallyAnn. Great to meet all of you. Excited to be walking through prompt learning with you all. Uh, I don't know if you got a chance to see Harnit's talk yesterday, but hopefully that gave you some good background on how powerful meta-prompting and prompt learning can be.

Uh, so my name's Fuad. I'm a Product Manager here at Arize as well. And like SallyAnn said, we like to stay in code. Uh, we'll be doing a few slides, then we'll walk through the code, and we'll be floating around, helping you guys debug and things like that.

Uh, my background is also technical, so I was a backend distributed systems engineer for a long time, so no stranger to how important observability infrastructure really is. Um, and I think it's an appropriate setting in AWS for that.

So yeah, excited to dive deep into prompt learning with you all. Thank you.

Awesome. All right, so we're gonna get started. Just give you a little bit of an agenda of the things I'm gonna be covering. Uh, so we're gonna talk about why agents fail today.

What is efficient prompt learning? I want to go through a case study, kind of show y'all why this actually works. Uh, and we'll talk about prompt learning versus GEPA.

I think everybody-- I had a few people come up to me over the conference about like, "What about GEPA?" Uh, we have some benchmarking against that, and then we'll hop into our workshop.

Um, but with this I want to ask a question, how many people here are building agents today?

Okay, that's what I expected. Um, and how many people actually feel like the agents they're building are reliable?

Yeah, that's what I also thought. So let's talk a little bit about why agents fail today. So why do they fail? Well, there's a few things that we're seeing with a lot of our folks, and we're seeing even internally as we build with Alex for why, uh, agents are ba-- breaking.

So, um, I think that a lot of times it's not because the models are weak, it's a lot of times the environment, uh, and the instructions are weak. So, uh, having no instructions, um, from their learned environment, uh, no planning or very static planning.

I feel like a lot of agents right now don't have planning. We do have some good examples of planning, like we have CloudCode, Cursor, those are really great examples, but I'm not seeing it make its way into every agent that I come across.

Uh, missing tools, big one. Sometimes you just don't have the toolsets that you need. Um, and then missing kind of tool guidance on, like, which of the tools we should be picking.

And then context engineering continues to be, uh, a big struggle for folks. If I were just to distill this out, I think it's, like, these three core issues of adaptability and self-learning.

Um, so no system instructions learned from the environment. We've touched on determinism versus non-determinism balance, so having the planning, um, or no planning versus doing, like, a very static planning.

You want to kind of have some flexibility there. And then context engineering. I feel like it's a term that just kind of emerged in the last, like, you know, six to eight months, but it's something that's really, really important that we're finding, you know, missing tools, tool guidance, just not having context or including your data and, and

not giving the LLM enough context there. So these are, um, kind of the core issues to distill, but I think there's one other pretty important thing, um, and that is kind of this distribution of who's responsible for what.

So, um, there's these technical users, your AI engineers, your data scientists, developers, and they're really responsible for the code automation, pipelines, actually, you know, managing the performance and cost.

But then we have our domain experts, subject matter experts, AI product managers. These are the ones that actually know what the user experience should be. They probably are super familiar with, um, the principles that we're actually trying to build into our AI applications.

They're tracking our evals, and they're really trying to ensure that the prompt is a success. So there's this split between responsibilities, but everybody is contributing. But then there's this difference, um, in terms of, like, maybe technical abilities.

And so with prompt learning, it's gonna be a combination of all these things, so everybody's gonna really need to be involved, and we can talk about that, uh, a little bit more.

So what even is prompt learning? Uh, I'm gonna first kind of go through some of the, um, approaches that we kind of borrowed when we came up with prompt learning.

So this is something that Arize has been really, really, uh, dedicated to doing some research. And so one of the first things we borrowed from, uh, which is reinforcement learning.

How many folks here are familiar with how reinforcement learning works? All right, cool. Um, so if I were to give, like, a really, like, silly kind of analogy, if we have a reinforcement model, uh, pretend it's like a, a student brain that we're trying to kind of, you know, boost up.

And so they're gonna take an action, uh, which might be something like, you know, just gonna take a test, take an exam, and there's gonna be a score. A teacher's gonna come through and actually, you know, score the exam here.

Um, that's gonna produce this kind of, like, scalar reward. Um, and, you know, pretend the student has an algorithm in their brain that can just kind of take those scores and update the weights in their brain and kind of like the learning behavior there, and then we kind of repeat the process.

So, you know, in this kind of reinforcement learning, we're updating weights based off of some scalars. Um, but it's really actually difficult to update the weights directly, uh, especially in, like, the LLM world.

So reinforcement learning isn't gonna quite l-- work that well, uh, when we're, we're doing things like prompting. So then there's meta-prompting, which is very close to what we do with, uh, prompt learning, but still not quite right.

So here with meta-prompting, we're asking an LLM to improve the prompt. Uh, so again, we use that kind of, like, student example. We have an agent, which is our student, um, and it's gonna produce some kind of output.

Like, that's a user asking a question, getting an output. That's our test in this example. And then we're gonna score- Eval is pretty much what you can think of there, uh, where it's gonna output a score, and from there we have, like, the meta-prompting.

So now the teacher is kind of like the meta-prompt that's gonna take the results, uh, from our scorer and then update the prompt space off of that. Um, but it's still not quite what we want to do, and that's where we kind of introduce this idea of prompt learning.

So prompt learning is going to take the, the exam, we're gonna produce an output. Um, we're gonna have our LLM evals on there, but there's also this really important piece, which is the English feedback.

So which answers were wrong? Why were the answers wrong? Where the student needs to actually study, really pinpointing those issues. And then we still are using meta-prompt, but we still are asking an LLM, uh, to improve the prompt.

It's just the information that we are giving that LLM, uh, is quite different. And so we're gonna update, uh, the prompt there with all of this kind of feedback, so from our evals, from a subject matter expert going in and labeling, and use that, uh, to kind of boost our prompt with better instructions and sometimes exams.

So this is kind of like the traditional prompt optimization, where it's like we have... We're kind of treating it like an ML problem, where we have our data and we have the prompt.

We're saying, "Optimize this prompt and maximize our, like, prediction intervals." Uh, but that doesn't quite work, uh, for LLMs. We're missing a lot of context. So what we've really found, um, is that the human instructions of why it failed, so imagine you have your application data, your traces, a data set, whatever it is.

Your subject matter expert goes in, and they're not only annotating correct or incorrect. They're saying, "This is why this is wrong. It failed to adhere to this key instruction.

It didn't adhere to the context. It's missing out," whatever it is. Um, and then you also have your eval explanations from LLM-as-a-judge, which is same kind of principle, where instead of just the label, it provides the reasoning behind the label.

And then we're pointing it at the exact instructions, um, to change. We're changing the system prompt to help it improve so that we then get, you know, prediction labels, but we also get those evals, um, and explanations on why they failed.

So we're just kind of optimizing more than just, um, our output here. And I think the really key learning that we've had is that explanations and human instructions, or if you're an LLM-as-a-judge, that text is really, really valuable, and I think that's what we see not being utilized in a lot of other prompt optimization approaches.

Um, they're either kind of optimizing for a score, uh, or they're just paying attention to the output. But you kind of think of it as this way. It's like these LLMs are operating in the text domains.

We have all this rich text that tells us exactly what it needs to do to improve. Why wouldn't we use that to actually improve our output? So, um, that's kind of the basics of prompt learning.

But everybody always comes up to me and is like, "Sounds great, Sally, but does it actually work?" Um, it does, and we have some examples of when we do this.

We did a little bit of a case study. Um, I think coding agents, everybody is pretty much using them at this point. There's quite a few that have been really, really successful.

I think Cloud Code is a great example, Cursor, but there's also Claude, uh, which is more of a, um, an open version of this. And so we decided to take a look and compare to see if we could, you know, do anything to improve.

So these are kind of the, the baseline of where we started here. Um, you can see the difference between the different models. Um, Claude was obviously using two and throttle, kind of the state-of-the-art there.

But we also had this opportunity where Claude was using, you know, four or five, Sonnet, and it was working seemingly well at thirty percent versus forty. Um, and then there was kind of the conversation around four point one.

So this is where we started. Um, and we took a pass at optimizing the system prompt here. So you can see this is what the old one was looking like.

It has, like, no rule section. So it was just very, like, you are a Claude agent, you're built on this model, you're, you're here to do coding. Um, but there was no rules.

And so we took a pass at updating the system prompt so that there were all of these different, uh, rules associated. So when dealing with errors or exceptions, handle them in a specific way.

Um, make sure that the changes align with, you know, the system's design. Um, any changes should be accompanied by appropriate tests. So really just kind of building in, like, the rules that, like, a good engineer would have, uh, which was completely missing before.

Um, and so we found that Claude performed better with updated system prompt. Pretty kind of simple. It's kind of the whole concept here is, like, you can see these different problems, and we're seeing, you know, things that were incorrect now being correctly done just by simply adding more instructions.

So really demonstrates pretty well here, uh, how those system prompts can improve. And we benchmarked again with a SBE Bench Light to get another just, like, kind of coding, uh, benchmark for these coding agents.

And we were able to improve by fifteen percent just through the addition of rules. Uh, so I think that that's pretty powerful. So no fine-tuning, no tool changes, no architecture changes.

I think those are the big things folks, like, reach for when they're trying to improve their agents. Uh, but sometimes it's just about your system prompt and just adding rules, and I think we've really seen that, and that's why we're really passionate about prompt learning and prompt optimization in general, is it feels like the lowest lift way

to get massive improvement gains in your agent. Uh, four point nine achieved performance near Sonnet four point five, which is pretty much considered right now the state of the art when it comes to coding questions, and it's two-thirds of the cost, which is always, uh, really a better visual.

So, uh, these are some of the, kind of the tables here. We'll definitely distribute this so you can kind of take a closer look. But I think the main point I want y'all to come away with is the fact that, like, you know, fifteen percent is pretty, you know, powerful, uh, improvement in our performance.

Now, a question we get all the time is we're taking these examples, uh, for prompt learning. So how does this really work? So we're gonna take a data set.

A lot of the time that data set is going to be a set of examples that didn't perform well. Either a human went through and, uh, labeled them and found that they, you know, were incorrect, or you have your evals that are labeling them incorrect.

And so you've gathered all these ex-examples, and that's what we're going to use to optimize our prompt. So I get a question all the time is like, "Well, aren't we going to overfit, uh, based off of these bad examples?"

But- There's this rule of generalization where meta-prompting enforces high-level reusable coding si-- coding standards rather than repo-specific fixes. Um, and we are doing this train test split, uh, to ensure that the rules are generalized beyond just like these local quirks and whatever our, uh, training dataset is.

But if you kind of think of this as, as like you hire an engineer, right, to, to be an engineer at your company, you do kind of want them to overfit to the database that they're working on.

So, uh, we kind of feel that overfitting is maybe... Uh, a better term for it is expertise. Uh, we are again not kind of training in the traditional world.

We are trying to build expertise. And as we'll talk about, this is not something we feel that you do once. You're actually gonna kind of continuously be running this.

So, um, more problems are gonna come up. We're gonna kind of optimize our prompt for what the application is seeing now. Um, and then we'll kind of continue. So we, we don't actually think it's a flaw.

We feel like it's expertise instead. Um, we can kind of adapt as needed and kind of mirroring what humans would do if they were taking on the tasks themselves.

Um, this is just another set of benchmarking, again, kind of proving here, um, that this diverse evaluation suite that focuses on the tasks for those difficult-- or tasks that are difficult for, for large language models.

Um, and we're seeing, again, success with our improvements. Now, GEPA just kind of came out recently, and I think that's something everybody's really excited about. I think the pr-previous, uh, DS5 optimizers were a little bit more focused on optimizing a metric.

And as we talked about, like, we really want to be using, uh, the tasks, the modality that these applications are working in, um, that have a lot of the, the reasons or how we need to improve.

And so we definitely wanted to do some benchmarking here. So how many people are familiar with GEPA or tried it, read about it? All right. Cool. Well, just to go like some high level, um, I just kind of noted that the main difference between GEPA and our other, like, MePro optimizers is that they are actually, um, using

this positive reflection and evaluation while they are, are doing the optimization. So it's this evolutionary optimization, um, where there's this parental-based candidate selection and probabilistic merging of prompts. What this really does under the hood is we take candidate prompts, uh, we evaluate them, then there's this reflection LLM that's reviewing the evaluations and then kind of making some

mutations, some changes, um, and kind of repeating until it feels like it has the right set of prompts. So I think something that is important to note about GEPA is it doesn't really choose kind of just one.

It does try to keep the top candidates, um, and then, you know, do the merging from there. But we benchmarked it, and Prompt Learning actually does do a little bit of a better job.

And I think something that's really key is it does it in a lower number of loops. And I think something that we'll, we'll talk about in just a second here is that it does actually matter what your evals look like and how reliable those are.

I think that's something we really feel strongly about at Arize is, uh, you definitely want to be optimizing your agent prompts, but I think a lot of people forget about the fact that you should also be optimizing your eval prompts.

Because if you're using evals as a signal, um, you can't really rely on them if you don't feel confident in them. So it's just as important to invest there, making sure you're kind of applying the same principles that your, your agent prompts as your eval prompts, so you have a really reliable signal that you can trust and

then feed that into your prompt optimization. But, uh, in both of these graphs, the pink line is Prompt learning. Uh, and we did also benchmark it against MePro, their older optimization technique that I was mentioning kind of options off, like, um, optimizing around score.

And evals really did make the difference. So kind of I, I highlighted on this slide here, like the pull with eval engineering, we were able to do this. So we did have to make sure that the evals we were using as part of prompt learning, uh, were really high quality because, again, it's-- this only works, um, if

the eval itself is working. So yeah, evals make all the difference. Kind of spent some time optimizing a prompt here. Um, again, it's all about making sure you have proper instructions.

Same kind of rules apply. So I want to kind of walk through... I know there was a lot of content. I feel like it's really important to have context.

But before we jump into any of the workshop stuff, any questions I can answer about what I discussed so far?

Uh, I, I have a-

Oh.

I have a question or-

Yeah

... just a general comment. So I, I think, you know, coding is the greatest example in terms of having the structure in evals. Uh, one thing I'm sort of curious about is if you have other examples, sort of general prompts for additional interactions with systems that are not as easily quantifiable.

I'm just curious about any experience you guys have there.

Yeah. Is that for like evals or just like the prompts in general?

Well, I think it's just clear how you would set up what the evals would look like. You know, I'm just wondering how you would do that for other types of-

Yes. Well, so the question is, like, is there any kind of instruction for how you should set up your evals? Coding seems like a very straightforward example. You kind of want to make sure the code's correct, right?

But where some of these other agent tasks, um, it's a little bit harder. I think the advice that I usually give folks is we do have a set of, like, out-of-the-box.

You can always start with things like QA correctness or focus on the task. But what I always suggest is, like, getting all the stakeholders kind of in the room, so getting those, you know, subject matter experts, PM, security, you know, leadership, and really defining what success would look like, and then start kind of converting that to different

evaluations. So, uh, I think an example is for like Alex, um, I have some task-level evaluation. So like I really care, did it find the right data, uh, that it should have?

Um, should it-- did it create a filter using semantic search or structured like making the right tool call? Um, and then I care, did it pull things in the right order?

Was the plan correct? So kind of thinking about like what each step was, and then like even security will be like, "Well, we care how often people are trying to jailbreak Alex."

So it's just taking each of those success criteria, converting it to evals. Um, and we do have different tools that can help you, but that's usually the framework I give folks is like start with just success and then worry about converting it to an eval after.

Yeah. Just to add to that, maybe like, um, more of like a subjective use case is like, for example, like Booking.com is one of our clients, and so when they do like what is a good posting for a property, like what is a good picture?

Defining that is really hard, right? Like- To you, you might think something is a very attractive posting for, like, a hotel or something, right? But to someone else, it might look really different.

And sometimes, as kind of SallyAnn was alluding to, it's sufficient to just gate it as a good/bad, and then kind of iterate from there. So like, is this a good picture, a bad picture?

Let an LLM decide, and then gate from there into specific fact. Like, oh, this was dimly lit, the layout of the room is different, et cetera, et cetera. Yeah.

Yeah, that's... That you're actually building on the question I was gonna ask, which is that if you end up with that binary outcome, which doesn't necessarily give you a gradient to advance upon, are you then effectively using those questions like digitally lit/not to, like, get, like, a more continuous space?

Is that right?

Right. Exactly, right. And then from there, as you get more signal, you can refine your evaluator further and further, and then use those gates. And you can actually put a lot of that in your prompting itself, right?

So yeah.

Yeah. Okay.

I have two questions, and I'm not sure if I should ask both of them, or maybe your workshop will answer it. One is about rules and the rules section or, like, operating procedures.

I'm curious how you, uh... Do you just continuously refine that in the English language and, uh, maybe reduce the friction of any contradictory rules? That's the first question. And then the other was, I would love to see the slide on evals, and if you could just say a little bit more on how you approach that, because my

issue in doing this work is, um, whether or not to have, like, an, a simulator of the product and then the simulator is evaluating, or to do what I'd like to do, which is, like, an end-to-end evaluation that I build.

But I would love to see you talk about that, if you could.

Yeah, absolutely. So sort of the first one about, like, how the instructions, it's definitely something I think that, like, you iterate over time on them. So a lot of times I think we take our best stab at, like, we write them by hand, right?

And I think what we're trying to do with prompt optimization is, like, leverage the data, uh, to dynamically change them. Uh, and LLM is, I think, great at, like, removing redundant instructions, things like that.

But the goal is, is we want to move away from static instructions. We feel very confidently that, like, that is not going to really scale. It's not gonna lead to, like, sustainable, um, performance.

So the idea exactly with prompt learning is something that you can kind of run over time. We see this even, like, a long-running task eventually, uh, where you're building up examples of incorrect things, uh, maybe having a human annotate them, and then the task is kind of always running, producing optimized prompts that you can then pull in

production and it, it kind of is, like, a cycle that repeats over time.

Sorry, just to intervene.

Yeah.

So are you saying that when you're doing this over a long period of time and then you have examples, you're just running the shots back into your rules section?

Kind of. You're just gonna pass it... Like, when we get to the prompt optimization, uh, actual, like, loop we're gonna build, you'll kind of see it as, like, you are feeding the data in that's going to build a new set of instructions that you would then, you know, push to production to use.

Okay.

Um, and then I think your second question was around evals and, like, how to... where to start, how to, like, write them and, like, how to optimize those. Is that right?

Yes.

Yeah, so it's a very similar approach. I think it's, like, the data that you're reviewing is almost a little bit different. So, uh, I should've pulled up the, the loops.

I don't know if you can find it. It should be

in this deck. Let me just try to find something really quick to kind of show this.

There we go. So this is kind of, like, how we, we see it, is you have two co-evolving loops. I've been talking about the one on the left, the blue one, a lot about we're improving agent, we're collecting failures, kind of sending that to do kind of fine-tuning or prompt learning.

But you basically want to do the same thing with your evals, where, uh, we're collecting the data set of failures, but instead of thinking about the failures being the output of your agent, we're actually talking about the eval output.

So having somebody go through and, you know, evaluate the evaluators or using things like log prompts as confidence scores or a jury-as-a-judge to determine where things are not confident, we're kind of doing the same thing.

So figuring out where your eval is low confidence, and then you're collecting that, annotating, maybe having somebody go through and say, "Okay, this is where the eval went wrong."

And so it's the same pretty much process of, of optimizing your eval prompt. It's just, you know, I think folks think they can just grab something off the shelf or write something once, and then they can just forget about it.

But this loop, I, I've said it a few times, but the, the left loop only works as well as your evals are working.

Sorry, I think my question is actually way more static and basic.

Oh.

It's like, do you... Are you talking about this orange circle as, like, are you building a system or simulator for the eval, or are you just talking about, like, system prompt, user prompt, eval?

Yeah, I think it's more... Right now what we're talking about is just, like, kind of the different prompts. You can definitely do simulations, but I think that's a, a whole different workshop.

Thank you.

Okay.

Any more maybe questions before we get into the workshop? Any?

Cool. Let me switch back. All right. Um, so here is going to be a short code, uh, for our Prompt Learning repo. Um, so I'll give everyone a few minutes to get set.

Should we do it with that? Uh, get it on your laptops. I know it's a little bit clunky. The, uh, [laughs] this kind of QR code, like, AirDrop it. I was not sure a better way.

Um, I can just show you also here if you want to find it. Um, it is gonna be in our Arize AI, uh, repo here, and under prompt learning.

And you just want to kind of clone that. We are gonna kind of be running it, uh, locally here.

Can you go back to the page with the URL?

Yes.

Okay.

Sorry about that.

No, the page with the URL.

Oh, sorry. I'll give folks just a few minutes to get... Yeah.

What do you... What's your process when you're building a new agent or workflow? Anything that could be evaluated, do you guys start by just like, "Oh, let's try something with the prototype and then see where it's bad, and then do evals"?

Yeah, I think there's different perspectives on this. Our, our perspective is eval should never block you. Like, you need to get started, and you need to just build something really scrappy.

We don't think, like, you should, you know, waste time doing evals. I think it's helpful to pull something out of the box sometimes in those situations, just because it's hard to comb through your data.

Like, that's something we've experienced with Alex, of like when you're getting started, just running a test, manually reviewing, like it's, it's kind of painful. Um, so I think that having evals is helpful, but shouldn't be a blocker.

Pull something off the shelf, maybe start with that. Then as you're iterating or understanding where your issues are, then you're starting to refine your evals as you're refining your feature.

Yeah. One last question.

Yeah.

So it makes sense to, like, optimize the system prompt. But can you use any, like, blog codes, sub-agents, or commands? Or how are you thinking about this, like, multi-agent?

Yeah. So the question is, is like you're just doing one single prompt or how do you think about this in a multi-agent? I think we're kind of thinking of this right now as kind of independent tasks that can optimize your prompts kind of independently into running tests, um, to get into, like, the agent simulation of running them

all together. But right now, our approach is a little bit isolated, but I definitely see a future where we're gonna kind of meet the, the standard of, like, sub-agents and everything else that's going on right now.

Yeah.

No, I think that's pretty accurate. And also, like, I mean, even in a single agent use case, uh, versus like a multi-agent use case, like, ultimately, like, each of those agents may be specialized.

They may have their own prompts that they need to learn from. So I think doing this in isolation still has benefits for the multi-agent system as a whole that can compound over time, especially like in scenarios like handoff and et cetera, and making something like really, really specialized.

So, uh, kind of gets to, like, what we were talking about with, like, the overfitting as well, which is again, like, the question we get all the time, but really you want to be able to fit on your code base as an engineer.

Um, you don't want to be so generalized that you're no longer good at thinking of specific works in your code base. But yeah.

Okay. Everybody kind of getting to the read-through okay? Anybody need any help?

Um, all right, so we are gonna be using, um, OpenAI for this. So I think the next thing that I'll have everyone do is probably spend some time describing your AI keyboard into it, and then I'll just kind of start walking through our notes here.

Okay. So we are gonna be doing a JSON webpage prompt example. So you're gonna find that under notebooks here. Um, and so we'll give everybody a second to pull that up.

There's gonna be just some slight adjustments we're gonna add to this example, uh, just to make it run a little faster and work a little better. So first is, um, what this is even doing.

This is gonna be a very simple example, uh, for just a JSON webpage prompt. If anybody has like a prompt or a use case that they want to kind of like code along, uh, Fuad and I are absolutely help-- like glad to help, kind of adapt what you're working on to the use case here.

So something very simple just to kind of demonstrate, um, the, the principles. And we are gonna be using Aura. We can definitely experiment. If you want to swap out any other providers that you want to use, we can also definitely help you do that.

Um, but the, the goal of this is essentially going to be to iterate through different versions of a prompt using a dataset, um, and we will optimize. So the first thing is obviously we need to do some installs.

Um, I am just gonna have you all update it. I think it says like greater than two point zero zero, uh, but we're gonna actually just use, I think, two two today.

And then the next thing is just to make this run a little faster. We're gonna run things in async, which is missing, uh, so you can go ahead and add these lines in the cell as well.

All right. Everyone kind of follow along? And I never know... I don't want to move too fast, but it seems to have not. Cool. Let's talk about configuration. So, um, I kind of talked about it a little bit when I was going through the slides, but we are going to be doing some bluebing.

So the general idea is, is we start out with a dataset, uh, with some feedback in it, and we'll, we'll look through the dataset once we get there. Um, but you're gonna want to have either human evaluation, um, so like annotations, either free text, labels, um, or you're going to want to have some evaluation data.

But the feedback is really important. That's what makes this kind of work. Um, we're gonna then, you know, pass that to an LLM to do the optimization, and then it's going to basically have evals in the loop.

So as it's optimizing, it's using that kind of dataset to then run and assess whether or not it should, you know, kind of keep optimizing. Um, and then it also provides you data that you can kind of like use to gauge which of the prompts that it outputs, um, in, you know, a production setting.

So we're going to do some configuration. Um, so I've kind of wrote out here kind of what each of these means. So we have, uh, the number of samples.

So this controls how many rows of the sample dataset. Um, you can, you know, set this here to use all data, or you can, you know, use a positive number to limit for, you know, faster experimentation.

So I think that sometimes folks use, you know, different, uh, approaches here. Sometimes you want to just move really quick, so you set a low sample. Sometimes you want to be a little bit more representative, so you up it.

Um, I have it here set as a hundred. Feel free to adjust. Um, and then the next thing is train split. Um, so I think folks are probably pretty familiar with the, the concept here of like a train test split, but it's just how much of the data do we want to use into our training?

Again, that's what we're using to actually optimize. Then how much of it do we want to use when we're testing, when we're running the eval, um, on the new prompt?

Um, then there's number of rules, um, basically the specific number of rules to use for evaluation. This just determines which prompts to use. Um, and so this is like as we're running these loops, we're outputting, you know, a bunch of different prompts, so this is just saying how many, um, we should use for evaluation.

And then key one here, number of optimization loops. So this sets how many optimization iterations to run per experiment. Um, and each loop ge- basically generates those outputs, evaluates them, and refines the prompts.

And so these just control the experiment scope, the data splitting, um, just pretty much the whole prompt learning loop and, and how much data we want to use. So you can kind of just- Run these as you are.

If you want to go through some, feel free. Uh, and then the next step, pretty simple. We're just gonna, uh, grab that OpenAI key if you haven't already, uh, set that up.

So get pass is just gonna, like, pop up. Um, I'll show you here quick. It's gonna pop up there. You can just paste in your API key. I'll say before we start looking at the, the data little bit.

And just if anybody runs into any issues, just give us a wave. Well, I think this part's the same, so hopefully we get through this way.

How are we doing on API keys?

Pretty good.

Let's go. Let's see.

Yeah.

I'm, I'm doing good, but if you have a free one you want to give me- [laughing]

-you can use that.

I wish.

All right. Let's talk about the data. So we provided the data with you with queries. Um, you can see here that we're doing the eighty/twenty split based off of kind of configuration we set above.

We're just gonna pull this, um, train set here, and let's just-

Yeah, I run the eighty/twenty because in the driving use minus other fifty/fifty.

Oh, yep, you're right. That's a mistake on my part.

Yeah, it is the fifty. Um, let's take a look at what this data set looks like. No. Uh, just so folks can kind of understand. Um, so kind of starting here with some just basic input and output.

Um, for initial train set, we don't have any of the, the feedback in these rows that I printed out here, but you can imagine you can have different, uh, correctness labels here, explanations, any rule violations.

This data can be whatever it is that, um, you'd like it to be. Some folks use multiple evals, some are just one. Sometimes it's human feedback, sometimes it's a combination.

Uh, but you really want to have, you know, the input and output that we'll use to evaluate.

So should my output, a train set, be the same as you?

Not necessarily.

Okay.

Depends on-

I, I didn't know if the head, if the head was sort of or not. Like I-

Yeah. It all depends on kind of what the, the sample. We could look at, like, you know, if I did this, this should be the same for you maybe.

Just sending sharing.

Yeah. I-- Yeah.

That's what you're saying? Okay. Yeah.

Quick question.

Mm-hmm.

Um, is it possible for the input to be like a chat history and not just a-

Quick question. So I think it depends on, like, what it is you're trying to do. If you're doing just like a simple kind of, uh, system prompt at the input, you kind of want it to be one to one.

You don't want to give it a ton of, um, like conversation data that is not relevant to the prompt that you're optimizing. Um, we, we generally just use like the single input, but I think that there are applications that you could do like conversation-level, um, inputs.

Yeah, be-because, because quite often the, the failure is somewhere in the middle of the conversation, right? So, so if you put just the original task in, uh, then, uh, the probability of you hitting, you know, a failure in the middle of the, uh, prompt, you know.

Totally. So in that case, what you generally see is like different rows of like having each of, like, the back and forth be like kind of independent rows, 'cause you're probably gonna evaluate each of them and, um, obviously probably like get the human feedback on each of them.

So we usually separate them out in that way. But it's a good point. Like, you just always are focusing on the first part. There's probably like a lot of redundancy there.

Um, you definitely want to like sample parts and parts of the conversation.

And, and how we can bifurcate like, uh, instruction versus we have some context also. So-

Yeah.

-it should not fetch the context. It should only, uh, whatever the manipulate the, uh, system instruction or the prompting. Context, it should be the static. It should not be like, uh, based on the answer it will change my context.

Yeah. So what you're saying is, like, with the input there might be like additional quality and context you're kind of passing in. You can absolutely include that in your data set, um, so that the application can understand what other...

Or not the application, but the prompt learning, um, LLM can understand all of the data that's kind of like available. So you can just have that passed in as extra column if you want.

Most people start with just kind of input and the feedback. Um, but you can absolutely add whatever data you think is relevant.

And when for the rerunning, when we're doing the experiment of testing, you'll definitely always want to have the data that would be required to answer.

Any agent, like even very simple agent has some tool call, some rec call, or some context it is pulling, uh, some API call. So whatever the prompt engineering, it should be based on the out, uh, after getting the output.

Right? So it can take whatever the context it contains my prompts, plus whatever the tool call I have done, API call, all the, uh, context engineering, and then last, uh, final answer.

Totally. Yeah. So a-again, we're-- at this point we're, we're testing just like one prompt and not the kind of end-to-end, but you definitely want to have everything that like is flowing into the prompt that you're optimizing.

So, uh, if your system prompt takes in the user input, for example, uh, some data from an external API, you definitely want to provide all of that data. Does that make sense?

Because the, because you're saying that like the, the like trajectories, the like tool calls and what the agent's gonna do will be depending on what the tool call was, is what you're trying to proper learn to.

Y-yeah, exactly. We wanna just like-- Because we're kind of trying to replay optimize one step of it, and we definitely don't want to do it in completely in isolation.

So if there's like data that flows into that prompt, um, that's context that's using-- that's producing the output, right? So we want to be sure that we're including that.

We don't want to exclude anything. But if it's data that comes like at, at a different step, probably not relevant. Then you don't want to do that that way.

It's just like, think about what's relevant for the, the step that we're trying to optimize and this decision.

All right. Any other questions on kind of that one?

All right, cool. So, uh, for the set of our initial system prompt, you can see this is something very, very basic. Uh, definitely I think we can do a whole lot better than this, but just kind of wanna illustrate something, uh, that we're going to test and optimize.

So we're just saying you're an expert in JSON webpage creation, your task is input. And then so all these inputs that we're seeing here are going to be what we're actually generating outputs for and trying to optimize for that.

Now, I already kind of touched on this. Um, evaluators are extremely important to make all of this work, right? Um, so we're going to, uh, introduce two evaluators that use LLM-as-a-judge to assess the quality of generated outputs.

So we're using LLM-as-a-judge. If you have any other, like, code-based evaluations, whatever you need to do to evaluate, you can definitely swap those out. Uh, but we're gonna do evaluate output.

This is gonna be a comprehensive evaluator that assesses the JSON webpage correctness against the input query and the evaluation rules. It's going to provide an output label of correct or incorrect, so pretty simple binary.

Again, you can use multi-label. And then it's going to have the detailed explanations as well. Um, and then we have a rule checker. This is a more specialized evaluator that performs a granular rule-by-rule analysis, um, and it examines if each rule, um, was compliant.

And then both of these are gonna generate feedback that goes into our optimization loop, uh, to iteratively improve the system prompt, um, explanation rule violations guide, um, and we'll get to this, the prompt learning optimizer, and creating the more prompts.

So, I have some imports here. Let's take a look at what the actual evaluate output has. Um, so we do have some rules that are in, um, in here.

Wait. Sorry. [laughs] I don't want to read anything wrong. Um, they're gonna be in the repo. Um, so we're gonna open that as a file. We have this LLM provider.

Again, we're using OpenAI here, and then we're gonna do our classification evaluator. So, uh, we're just calling it, uh, evaluate output. Um, it's LLM. We have an evaluation template that we're reading from the file here.

Um, then we just have choices, correct and incorrect. Now, we're mapping a label to a score. Sometimes it's helpful to be able to, like, add or to score. Sometimes a number is easier than just looking at a bunch of labels.

Uh, it is optional if you want to map these. If you have like a multi-class use case, you can set the scores, uh, accordingly. But these are just gonna be our choices, like the rails that we want our LLM-as-a-judge to adhere to.

And then all we're doing here is getting our results. See if I have it doing some printing so you can kinda take a look. So this is gonna be slightly different than what you're seeing in the notebook, so I'm just gonna pause here.

Uh, if you want to make the code changes from what you're seeing in probably your version, this is a, a good time for that.

Just kind of the setup of the evaluator makes sense to y'all? Like, kind of the key points? It's gonna be the rails, it's gonna be the output, uh, and of course our template.

Do we need to use our own?

Yeah, you will want to grab your own, uh, OpenAI key, uh, here, uh, to set.

Okay.

And we can help you. If you want to use a different provider, we can help you swap this out quick if that is helpful to anybody.

Okay. I'm gonna start walking you through the output generation. So, uh, this is just kind of, you know, you can imagine this as your own agent logic or the, the part that you're kind of testing.

Uh, this is just gonna provide a function that actually generates the JSON, uh, outputs. We're using four one here with JSON, uh, response format. So your temperature for consistent outputs.

Um, it's taking a dataset, a system prompt generates outputs for all rows, returns the results for evaluation, um, and it's called during each iteration to produce output. So this is like our experimentation function that we're writing.

So as we're passing in data, producing new f-- uh, prompts, we need a way to test it, evaluate, understand, uh, how we are kind of moving the needle here.

So that's all this is. So it's a pretty straightforward function just called generate output. We have that output model. Again, we're using OpenAI. If anybody wants help switching things around, happy to help.

Uh, we are using response format because we are dealing with JSON here. So, uh, we know that when you just prompt-- I mean, some of the, the newer models are decent at it, but, uh, using response format is really helpful, and then we're also setting temperature to zero.

Um, and then here is just kind of where we're passing all the data in. So the dataset, 'cause again, we want to run this on all of the, the testing data.

The system prompt, that will be input. So as we get to the optimization loop, we're gonna be passing in a new prompt to this with the dataset and then evaluating.

Um, we have our output model that we've already passed, concurrency, all that good stuff, and it's just returning all of the outputs there.

Would you-- for the, uh, the current generation of models, since this one's basically like in, in AI terms, ancient, uh, would you, like, still recommend setting the, the temperature to zero, or would you actually want to try to encourage some of the creativity into it?

So, like-

I think it depends on the use case a little bit and what you're, you're trying to do. You could definitely experiment that and kind of take it to the lens of how, how important is consistency to you.

Something like, I feel like JSON webpage, I feel like-

Yes

... consistency probably, like, temperature zero makes sense, but I definitely think not for every agent, every use case do you want to use zero.

Any other questions? Okay, moving. All right. Additional metrics. So we kind of talked about before that we are kind of using some score routing. Uh, this part is optional.

You want to use the metrics that make sense to you. We're not directly using this, um, as like we are kind of like using it to know whether or not we optimize, but it's not like we're, you know, using this as our sole kind of indicator for the success.

Uh, here we are just going to calculate some very basic metrics. Um, it's just you can, you know, choose something like accuracy, F1 precision recall, just some basic kind of classification metrics for us to understand.

Again, 'cause we are using binary mapping to scores, we can do that. Um, and so that's what you're seeing happen here. We're mapping to binary, and then just based on the score, we calculate the metrics.

So very super simple, uh, helper function here.

All right, the good stuff, the optimization loop. We made it. Um, okay, so this cell implements the core prompt optimization algorithm. It's a three-part process. Uh, so we want to generate and evaluate.

So generate outputs using the current prompt on the test dataset and evaluate their correctness. Uh, we want to train and optimize. If results are s- unsatisfactory, generate, uh, outputs on the training set, evaluate them, use the feedback to improve the prompt, and then iterate.

So we kind of want to repeat until either the threshold is met or all the loops are rep- um, kind of completed. So if you remember above, um, we're kind of setting that to just, like, five loops.

Um, and then, you know, we can kind of repeat, um, based off of that or if the threshold's met. Um, it's going to track metrics across all the iterations.

We'll turn to detailed results, including a train test accuracy scores, the optimized prompts and the raw output. So as I kind of mentioned at the beginning, as we're running these different loops on the experiments, we're gonna be producing a lot of different prompts.

Um, and so we're kind of getting that information back that you can use. Um, and then these are our key parameters. I'll kind of go through them, you know, as we get to the code, but just to give you a heads up, uh, this is the target accuracy score to stop optimizations.

Um, it could also be whatever other metric. You'll see we have a score, so you can kind of determine. The number of loops of the optimization iterations, we've set that score.

And then the number of rules. Again, these are some configurations we've already set.

Um, cool. So optimization loop. This is, um, going to take in all of those, um, you know, parameters that I've mentioned there. Um, it just kind of kicks off saying, "Hey, we're starting."

Um, it's going to do the initial evaluation so we understand, uh, how things are starting off. Uh, yeah, you can kind of pass in data too. You can kind of skip this initial evaluation.

We're kind of running it, uh, at the start here, but if you were running a production setting, you might already have evals, you might already have feedback. You can kind of adjust this for that.

Um, and then it's going to assess the threshold against kind of our initial evaluation. Again, this could kind of be skipped when we're coming from a production setting, but wanted to kind of start us off from scratch so that we can get a real feel for this.

Um, and then it starts the loop. So we're generating output. Um, it's setting that as the train output. So when I printed train, you kind of saw the outputs.

I kind of skipped ahead there. Um, and then it also will set, um, you know, correctness, explanation, any rule violations. Um, and then we'll actually use our prompt learning optimizer.

So this comes with, like, the SDK, uh, the prompt learning SDK that you can use, um, with Arize. Um, so we're sending in that prompt optimization, uh, the prompt choice, um, and then that API.

So under the hood, as we talked about in the slides, taking in that feedback, um, taking in the original prompt and trying to optimize to get better results and then spitting out the prompt.

Um, and then can also add in evaluator. So again, those three, um, kind of feedback columns we're looking to get back is correctness, explanation for that if there are any rule violations.

And then from there, we just kind of kicked off the optimizer and optimized with our train set. Output those feedback columns again and then, you know, any context size limitations you want to add.

Um, next step, so the optimizer again is gonna take our data, produce a prompt we want to evaluate so we understand how we're doing, what this code block doing-- is doing here.

So trying to get that new prompt running it again with all of those evals, getting our result okay. And then we do that, uh, with our test set as well.

And then we're getting back, like, our score and our metric value, and then doing the checks, and then we repeat it all again till we either get above our threshold or we've hit the max number of loops, and then returning our results.

So that's kind of what's going to be happening, and I want to pause here. Any questions on that?

Um, just some results saving functions and more helper functions here. So we do want to obviously save all these results. We don't want them to just be ephemeral and we can't ever access again, so just saving them all.

Um, you can also save all the single experimentation so you have all of that data towards the end, where we'll kind of pull this, um, and determine what the best prompt is, but these are just very basic helper functions.

I won't spend too much time, just saving them to CSV at the end of the day.

Now we use the execution bit. Um, so this cell runs the prompt optimization experiment, saves the results. We're getting the JSON format, the CSV format. Um, it includes columns for the iteration number, the number of rules, test training accuracy scores, all the data that we're actually gonna need to evaluate, uh, whether or not this thing is successful.

And then we're gonna start getting, uh, results here. So, um, this does take quite a while to run, so we'll run, and I think this will be a great point for a discussion.

But as we kind of are running it, you're gonna start seeing the different loops, um, kind of outputs coming out as well. Um, and yeah, we'll just kind of, like, work through it as it, it runs.

It's probably gonna take like twenty, thirty minutes for things to run. But, um, happy to take any questions and help anybody out as they run into issues.

Small thing. Can you scroll back to the part of the code that we needed to change?

Oh, yeah.

Yeah. Thank you.

Then change something.

It's gonna be there.

It's not...

Change everything.

Yeah. Yeah. Yeah.

So one reminder, um, so I know a few folks are running into this. I don't think I was... So this line here, when you're doing your pip install, you do want it to be equals equals two point two, um, 'cause I think there's a, a little bit of a package issue.

Um, so just make sure that's there if you're getting errors with the eval, it's probably why. If not, let me know and I'll come try to fix it.

This is the reason why-

Uses, like, a generic, uh, evaluation prompt right now.

Yes. And you can kind of see the evaluation prompt if you go to the-- We've kind of just taken that part out of this, but we can definitely go through that.

Um, so if you look here, um, on this line here, we're reading in, um, under prompts here

You can find the evaluation if you're curious.

And, and this is the reason why everyone hates on Docker, but this is why we use Docker. [laughing] Actually rerun the whole machine.

Yes, absolutely.

Don't work a lot. The Notebook happens with Arize script.

So I would also recommend, uh, patching your code with nest_asyncio, if you haven't already. Helps it run a lot faster. Also, for the purpose of the workshop, um, I switched our loops to one, uh, that took me six minutes to run, so I would recommend also doing that instead of having five loops.

Obviously, you wouldn't recommend doing that when you're actually trying to optimize your prompt, but for now it'll help you get through the workshop.

All right. I just wanted to kind of call out the, the last little bit here. Um, the last set. Yes.

Before folks head out. Let's see. Okay. Um, so the, the last little bit of code here, um, is just to extract the prompt that achieves the best test accuracy.

So I've mentioned how we're kind of like saving out all the results to use. Uh, we just have a function that es- essentially gets the last or the best, uh, version of that, kind of showing you the original and then the best optimized version, uh, which you can then use to kind of pull and put into your,

um, code. I did want to kind of just give one kind of call-out. Um, as you kind of saw today, doing this all in code can be a little bit, um, difficult to, to manage.

And so I want to call out for those of you who are kind of maybe looking for more of like an enterprise solution to this. In Arize, uh, you do have these prompt optimization tasks.

Uh, you can have your prompts living in our prompt hub, um, datasets with all of your human annotations or evals that you can either create from traces or just by ingesting it into Arize.

Um, and then from there, all you really need to do is, like, give it a task name, choose what you want your training data set to be, where the output lives, where all your feedback columns are.

Uh, you can adjust all of the parameters, uh, that you'd like. And then from there, you can just, like, kick it off, and it will produce an optimized prompt in the hub for you.

Um, so if I go over here, I think I have some... No, maybe not. Uh,

it will basically just create a new version here that says it's optimized prompts with all the results, and we are building on this, so you can add all your evals to it, have that all running in the loop.

Uh, but just wanted to call out that if you're not interested in maybe maintaining code loops and having to build, uh, like a task infrastructure yourself, it is something that we do offer in Arize.

Um, but yeah, hopefully some folks are heading out. Uh, we'll be sticking around here for a little while as we, um, can help you kind of work through issues.

But, uh, thanks so much for joining us. Um, hopefully you have learned something useful. [upbeat music]