[Evals Workshop] Mastering AI Evaluation: From Playground to Production

[upbeat music] Hey, everyone.

Thanks for joining us for the eval session today. This is the first workshop that we'll be leading. There's another one at 3:30, so you get to be the first people to, to go through it.

Uh, very exciting stuff. If you've gotten a chance to sign up for Braintrust, uh, you know, please do that now. If not, we also have some workshop materials and a Slack channel for you to follow along.

In the Slack channel, we also sent out a poll. If you'd like to respond with a little emoji underneath the message, that'd be great.

Uh, in the Slack channel also, there is the, the workshop guide. So in, in case you're not, uh, able to get the QR code for whatever reason, uh, go into the Slack channel and you, you'll be able to pull up that document.

Before we jump in, obviously maybe just a quick intro of, uh, Carlos and I. My name is Doug. I am a solutions engineer at Braintrust. Um, I have a background in data and finance.

Actually, my, my third week here at Braintrust. Um, but looking forward to kind of leading you all through the platform and giving you a sense for how you can master evals with Braintrust.

Yeah. My name's Carlos Esteban. I'm also a solutions engineer helping out with some of our great customers at Braintrust. I'm a little bit more tenured, been here six weeks, and before I was in the info world working at HashiCorp doing some, uh, stuff with Terraform and Vault.

Uh, but yeah, super exciting to be here today at the AI World Fair. Uh, we have a lot of exciting things to go over with you.

So just to go over the high-level agenda, we're gonna be alternating between lectures with slides and hands-on activities. Uh, so we're gonna start with understanding, you know, why even eval, what is an eval, uh, go over the different ingredients or components, if you will, there.

Uh, then we'll jump into the actual Braintrust UI. You'll go through some activity, uh, tasks there, and then move back into the lecture, talk about the SDK, how you can do the same thing via the SDK.

It can be a bit more powerful in certain situations as well. Uh, then go into, uh, production, like logging, so day two stuff. How are you-- how are you observing your users interacting with your production app or production feature?

And then finally, we're gonna be incorporating some human-in-the-loop, so trying to, uh, establish some ground truth for the ideal responses, improve your datasets, and overall improve the performance of your, of your app.

If you've gotten a chance to check out the poll in the Slack, uh, feel free to submit a response. Uh, really curious to see how everybody is currently evaluating your AI systems.

Just as a, you know, question that I'm out of curiosity, could I ask for a, a show of hands how many people have seen Braintrust before, gone to braintrust.dev and interacted with it?

Cool. That's great. So we have some, some, uh, pupils that have already gone and, and explored a little bit. That's exciting, and a lot of people that are brand new.

So starting off with just an introduction. What are evals? Uh, how do you get started?

First, I wanted to just show off some mentions of evals in the public space. Uh, you may recognize some of these names. They see the importance of evals, which, you know, may or not, uh, point you that, that this is something important that we should be thinking about when pushing changes into production, when we're developing these, these

AI features. So why even do evals? Well, they, they, uh, they help you answer questions. That's ultimately what they're for. You know, uh, what type of model should I use?

What's the best cost for my use case? What's going to perform best in all of the edge cases that my users will be interacting with?

Is it gonna be consistent with my brand? Is it going to, uh, you know, uh, talk to the end cu-customer, to the end user in the same voice that I would want a human?

Am I improving the system over time? Am I able to catch bugs? Am I able to troubleshoot effectively? So all of this can be, can be answered with the help of evals, which is what we'll be discussing today.

The best LLMs, uh, don't always guarantee consistent performance, so this is why you need to have a testing framework in place, right? We have hallucinations, uh, occurring at a pretty high rate.

Performance is also degrading when you make changes. It's difficult to guarantee that the change that you're putting through isn't going to regress the, the application. And, you know, the changing a prompt, even if it may seem like it's improving it, will actually regress it.

So you need to have some, uh, scientific, empirical way of testing these changes and making sure that your AI feature is performing at the level that your users expect.

So how do evals help your business? Well, they cut dev time. You'll be able to push changes into production a lot faster. Uh, evals will live at the center of your development life cycle.

They will reduce costs. As the-- due to the automated nature of evals, you'll repla-replace manual review. Uh, it will then lead to faster iteration, faster releases. You'll also be able to optimize the model that you're using, make sure that it's the best bang for buck.

Your quality will go up, and you'll be able to scale your teams. It will enable non-technical users and technical users to also have a say in the prompt choice, in the model choice, and just the overall, um- Management of the performance in, in the production traffic.

These are some of Braintrust's customer outcomes. So we've been able to, uh, help some of these great companies move a lot faster, increase their team productivity, and increase their AI product quality.

So now moving into some of the core concepts of Braintrust. Uh, so we're really targeting three things. Prompt engineering, right? So we're thinking about how we're writing the prompts, what's the best way to, uh, provide context on our specific use case to the prompt so that we are optimizing its response, right?

Uh, the middle piece, evals. Are we measuring improvements? Are we measuring re-regressions? Is this being done in a statistical way, uh, that's easy to review, easy to understand? And then finally, AI observability.

Are we capturing what's happening in production? Do we know if our users are happy with the outputs? Unhappy? Are we able to, uh, prioritize certain responses so that we can keep iterating, keep improving?

Great. So now moving, uh, to the eval section. So what, what is an eval? So the, the definition we've come up with is that it's a structured test that checks how well your AI systems perform.

It helps you measure quality, reliability, and correctness across various scenarios. Ideally, you're capturing every scenario that a user will live through when interacting with your AI feature.

When it comes to Braintrust and writing evals, there's really three ingredients that you need to understand to be able to, to work effectively. The first is a task. So this is the thing that you're testing.

Uh, this is the code or prompt that you want to evaluate. It can be a single prompt or a full agentic workflow. The complexity is really up to you.

The one requirement is that it has an input and an output.

Then we have our dataset. So this is the set of real-world examples or test cases that we want to, uh, push through the task to see how it performs.

And then the score, that's the logic behind, uh, the, the evals. So how are we grading the output of our prompt on our dataset? Uh, these can be LLM-as-a-judge scores, or they can be full code, uh, functions.

And the caveat is that they need to output a score from zero to one, which will then be converted into a percentage.

I have a question.

Yeah.

Uh, what are the test cases? Is it also LL- agent generated?

It can be at first. The question was, is the dataset synthetic? Can it be synthetic? And the answer is it's a great way to get started quickly, is having an AI generate those initial use cases.

But as you progress, as you mature, it's great to ground those in logs, so you're capturing the real user traffic, the real interactions that users are having, and integrating those into your datasets.

Okay.

Great. So now I wanted to talk about offline evals and online evals. So there's two mental models to think through. Offline evals are what you're doing in development. Right.

So this is the structured testing of the AI model, of the, the prompt, uh, that you are going to then eventually, uh, show off to, to customers in production.

So this is for proactive identification of issues, right? Uh, this is what we'll be doing today in the playground in Braintrust and also via the SDK. Uh, but then on the other side is online eval.

So this is in production, uh, real traffic is being captured and being measured. It's being graded just like your offline evals are being graded. And this is going to allow you to diagnose problems, monitor the overall performance, and capture user feedback in real time so that you can understand, oh, this edge case isn't included in my dataset.

This is a weak point in my current AI product. I need to spend some time, uh, attacking it and improving it.

A big question that we get asked is, what should I improve? Right? Uh, I have my prompt, I have my evals, you know, how do I know what's wrong?

Uh, and I think this matrix really helps simplify this question. So if you have a good output from, you know, your own judgment looking at what the LLM is giving you, and it's a high score, then great, right?

You've verified yourself that the output is, is high quality, and also the, the scores, the evals have also come to the same conclusion. If you think it's a good output, but it's a low score, then that's a signal that you need to improve your evals.

Maybe the score isn't actually representing what a human would think, right? Uh, if it's a bad out- output, but a high score, same thing, right? It doesn't match what a human would think looking at the output, so you need to improve your evals.

And then finally, if it's a bad output and a low score, your evals are working correctly. That's good. And now you need to focus on improving your AI app.

So I hope this helps explain how you should be thinking through, uh, your scores and, and what to, what to tackle in which moment.

So now we're gonna zoom in to each of those ingredients or components, starting off with the task. Uh, so as I mentioned, right, a task is really just an input and an output.

It can be a single LLM call or a whole agentic workflow. It's really up to you what you want to test. Uh, so in this pattern, we're just going to be creating a simple prompt.

Uh, this is what the activity today is gonna encompass. And, uh, you can use dynamic templating with Mustache. So you can provide your dataset rows as part of the, the prompt, and that will be tested, uh, and you'll get to see that in action soon.

What if you have more than just a prompt? What if you have a multi-turn chat, a whole conversation that you wanna evaluate? So you can do that in Braintrust today.

You can provide the whole conversation as extra messages, so providing that whole chain of, of, uh, messages back and forth with the user and the assistant. Uh, you can include tool calls as well to simulate those tool calls and evaluate that big chunk, that context, that whole conversation at once.

Tools are also something supported in Braintrust, so that's oftentimes something that your AI applications will leverage, uh, talking to external services or grabbing information from somewhere else. Uh, so you can add tools to Braintrust and have the tool available for your prompt, uh, to use.

And just to mention, that's great for RAG use cases. So I know that's a, a hot word right now. So if, if you have that in, you know, in mind, Braintrust can handle it.

We support tools, we support RAG. Um, agents, another hot word right now. Uh, so we also allow you to chain your prompts, right? So you can have three prompts chained together.

The output of the first prompt will become the input of the next, and so on. Uh, so you can start, uh, testing end to end, uh,

all these prompts back and forth, right? And, and do the same thing that you would with, with a single prompt.

Great, so now moving into datasets. So this is the test cases, right? You're gonna keep iterating over time, but initially, maybe you're using something synthetic. Um, there are three fields that you need to understand for a dataset.

Only one of them is required though, and that's the input. So that is the, the promp- the user-provided, uh, u-use case. The, the prompt that would be provided by, by the user would be the input, you could think of it that way.

And then you have the expected column, which is optional, which is the anticipated output or the ideal response of that, that prompt. Uh, and then finally, you have your metadata, which can allow you to, to capture any additional information that you may want to associate to that specific row in the dataset.

Some tips for datasets is to start small and iterate. Uh, y- it doesn't need to be the largest dataset of all time. It doesn't n-need to include all of your use cases, right?

Just get started. Use synthetic data at first. And the important piece is to keep improving, right? Keep iterating. So if you start logging your real user interactions, you know, even if it's just in staging or internally in your organization, you can start to increase the, the ca- the, the scope of the dataset, and it will start to

become closer to the, the overall, uh, domain of, of use cases that, that users will, will interact with. And then finally, you want to start implementing human review. Uh, this will allow you to establish ground truth, improve your dataset, improve the expected column, um, which will be great for, for your evals.

And zoo-zooming into scores. So this is, uh-- You have two options here in the type of score that you wanna use. LLM-as-a-judge, this is great for, uh, more subjective or contextual feedback.

What would a human need to understand when looking at the output? What criteria would they consider? Uh, this is more of a qualitative question that you want an answer to, right, using that LLM-as-a-judge.

On the code-based score, this is deterministic, right? So you would want, uh, exact or binary conditions. This is more of an objective question, um,

and it's-- The important piece is to try to use both. So you want some LLM-as-a-judge scores, but you also would like some code-based scores, and they, they'll help you meet in the middle and understand the, the quality.

So some tips here. Uh, you know, if you're using an LLM-as-a-judge, maybe use a, a higher quality model, a more expensive model to, to grade the, the cheaper model.

Uh, make sure that the LLM-as-a-judge has a focus. So don't give it, you know, four or five criteria to consider. Uh, zoom into one specific piece and expand and explain, uh, the steps it should think through, uh, to, to come to its conclusion.

If you're writing LLM-as-a-judge, maybe you should eval the judge and make sure that the prompt that you're using is matching what a human would think. Uh, so that's another great way of improving your scores.

And, you know, just make sure that it's confined and you're not overloading it with all the context in the world, right? You want it to be focused on the relevant input and output, uh, for consistency.

Great. Almost at the end here. So the-- there's two things to understand about the Braintrust UI specifically. So there's the, the Playgrounds, and this is for quick iteration of your prompts, uh, agent scores, datasets, right?

Uh, it's really effective for comparing. Uh, you can do A/B testing with prompts. You can do A/B testing with, with models. And then you can save a snapshot of the playground to your experiments view.

And the experiments is for comparison over time. Uh, so you'll be able to track how your scores change over weeks, months. Uh, and everything that your team is doing across the UI, across the SDK will also aggregate in the experiments view.

So you can analyze everything and understand, okay, this new model came out today. How is it performing to the prompt from two weeks ago?

Great. So now we've reached the first activity. Uh, so if you could please go to the activity document, and it will take you through the journey of, of running your first eval in the Braintrust UI.

Please raise your hand if you have any questions or run into any issues.

We'll be walking around and just, uh, making sure there's no blockers.

Do we have the register document or is-

Yeah, if you check Slack, uh, we'll also go back to the QR code

So did every- did everybody have a chance to get these QR codes? Um, the, the middle one is gonna be the most important. This is where you're gonna access the materials for the workshop.

Can you probably upload it to SlideDeck please?

Yeah, I'll repeat the question. The, the question was around, uh, extra messages in the prompt, and if you are overseeing agents and, you know, multiple types of users, multiple different roles are, uh, all talking back and forth, and you wanna distinguish th- their roles, right?

And all being all within the, the playground UI. Um,

so you can-- right now, there is no additional delineation between, uh, the assistant, the user, the tool call, and the, um,

I believe that's it. So having the, the user be branched and it play different roles is something that you would need to rely on the, the SDK for that additional flexibility.

That was me. That was supposed to be my other question. You have the API, right? So that-

Right

... it can do things out of the UI programmatically.

Yeah, and, and we'll cover the SDK in the-

Mm-hmm

... in the next section. I think maybe the biggest takeaway is that there is no limit really on the complexity that, that you feed to that, like, as that task, right?

The, the only requirement is that input and that output. Like, maybe something is a little bit more tailored to the Braintrust playground and the UI, where some things are actually a little bit more tailored to that SDK.

Uh, so that's, that's-- we can jump into that in that next section. Um, maybe it makes sense to like, as we're going through, like, the, the, the workshop, I'll kind of walk through this as well, just so you can all kind of see me go through it.

But feel free to raise your hand. Uh, we can, we can walk around and, and answer questions.

Sorry, we wanna get this.

For Slack.

Can I take your-

Are you able to access the document via the-

Is the-

The-

The SlideDeck's not public? Is that-

It's not public.

Oh, the-- for the SlideDeck? Is this-

Yeah. We've, we've just-- Yeah, the question was this is all in the UI, right? Is this-- That, that, that's the only place we've been thus far, right?

Mm-hmm.

Uh, just talking a little bit about the, the different components of the Braintrust platform.

Um, so let me, let me walk through that, right? And give you a sense for what we just kind of showed to you in slides, right? Um-

So you can't access the SlideDeck?

We, we can, we can update that.

Yeah.

Yeah. Um, just let-let's kind of walk through this, uh, so we can get a sense for what, what we're, what we're building here. Uh, some of the things that, that Carlos just walked through.

Um, I have a lot of this stuff already installed. I hope that you kind of walked through this, right? We need, uh, certain things on our system to actually go and run this.

So we have Node, we have Git. Um, we're gonna sign up for a Braintrust account. Creating a Braintrust org. Already done this, so I'm not gonna kind of bore you through that, that step right there.

This project, unreleased AI. Um, if, if you, if you don't do that, you'll see two projects in your account. But, um, just this is where we're gonna actually create our prompts and our scores and our dataset from the repo that we're gonna clone into, uh, our local machine.

A part of this demo requires an OpenAI API key. That's just what we're using under the hood. That's certainly not a limitation of Braintrust. Um, you can use-- and maybe just to kind of highlight something here,

you can, uh, you can use really any AI provider out there. So y- if you've gone into Braintrust account, you've probably seen this. You've entered your OpenAI API key here.

This is what is gonna allow you to, to run those prompts in the playground. You can see you have access to many other providers. You have access to, uh, cloud providers like, uh, Bedrock and, and so on.

And then you can even, uh, use your own custom provider. But for this workshop right now, we are using OpenAI.

Question. Are you able to do this local, um, with local models local?

Sorry?

Are you able to run evals of Braintrust, uh, locally with local models?

Yes. Yeah.

We can.

The question was, "Can you run Braintrust locally using local models?"

Yeah.

Yeah. Um, we, we have, uh-- If you look a little bit further out, there's a section for what we call remote evals. Might not have time to get to it in this particular section, but know that you can go to that and, and play with that feature as well.

Um, sorry, coming back down here. Um, so we're gonna clone this repo, right? This is, uh, right, this is the application that we're creating. The idea is to, uh, give it a, a, a GitHub URL and look for most recent commits since the last release, and then summarize those, right, for us as, as developers.

So that's the application that we're gonna use. Uh, we're gonna create some different API keys locally. So if you've cloned your repo, uh, you'll have a .env.local file. Uh, I'll show you my example.

You're gonna also input these, your, your Braintrust API key here, and then your OpenAI API key. This is optional down here. Uh, it's just if you don't wanna get rate limited by GitHub.

Probably not, probably not gonna create a lot of requests right now, so you probably don't need this.

Um, really important step here. So I'm gonna come back into, uh, Braintrust.

So as part of our install, we're, we're actually gonna go create some of these resources, uh,

within our Braintrust project that we just created. So I'm gonna run pnpm install.

This will actually go push some of these resources, and you'll find these, uh, in the Braintrust folder, the resources, and we'll jump into that, but just wanted to highlight that.

So now if I look back into my project, I should see that unreleased AI and the different things that we've created. We have two different prompts now, right? These are the, the prompts that we use to generate the change log, as well as the, the test cases, the dataset that we'll use as, as part of this

Yeah Uh, real quick, could you make the font a little bigger? Yeah, yeah, of course.

Um, code or here within the doc? Both? Uh, both, yeah.

Yeah. All right, let me stop there. A- anybody having issues just kind of going through that initial setup phase?

Where are the, where are the slides?

Excuse me?

Where are the slides?

Uh, we haven't made those public yet. I think we're-

Yeah, I'm trying to-

We're working on that

... to do that now.

Yeah.

So am I supposed to just take a QR code on my phone and then send a link to it?

Do you have, uh, do you have Slack? Uh-

I, I do have Slack.

Were you able to join the workshop evals channel?

The Wi-Fi is just, like, terrible. I can't really... I don't know. It's, like, on, um-

Same.

Yeah, the, the Wi-Fi's not working.

It's really slow.

It's slow.

Okay. Well, I'll, I'll walk you through. Yeah.

How are we connecting the clone repo to the, to the project UI if I have it like that? Yeah.

Um, yeah, so th... when we ran pnpm install, we ran a, a script in the background called just Braintrust push, and if I look at, uh, that file here, there's different things that we've configured, right?

We've configured, uh, my change log. So this is actually the Braintrust, uh, SDK under the hood. Uh, this is where we're creating that prompt in that project unreleased AI.

And so th- there's a couple things that we can do here from an S- SDK perspective. This is like, you think about, uh, you know, version controlling all of these different things and actually, uh, pushing them into the Braintrust UI.

So there, there's a lot of different ways to work with Braintrust, I think we mentioned earlier, either via just like the, the UI or actually via the SDK. But that's, that's how a lot of this stuff got created.

Okay.

Cool. Let's, let's kind of walk through this, uh, this first activity. Um, we're gonna, we're gonna access the unreleased AI project. So if we go to the prompts, so this is what we just created.

Uh, we created two different prompts, right? This is essentially what we can start to play around with, right? We have this one, uh, and you... and Carlos mentioned earlier there, there's this mustache syntax we can actually input, uh, variables here into, into our prompts, and this is gonna actually map to the, the different data sets that we

can actually use as part of this project. So here's our first prompt. It's about impossible to see. That was impossible? Okay. Well, it's- No, no, that's fine. I appreciate that.

Thank you. [laughs] And the lighting is hard, so-

Yeah.

Oh, maybe we can-

Change the appearance to light mode? I don't know.

Yeah, where's that?

Try with the user appearance.

How's that?

Oh.

It's daytime.

Thank you. No, I appreciate that. How's this?

Looks good.

Okay, cool. Um, so really just reviewing the stuff that we created. We created these two prompts. Here's our data set that we're gonna use when we run our evals and our experiments.

Uh, you can get a sense for, for this. Here's my input. I have a series of commits, and then I have a repository URL, and I have the, the...

when the, the last release was. The, that's that since field. So these, this is, again, the, the thing that we'll use inside of that playground to create, uh, to create evals and, and to use to iterate from.

And then the last thing that, uh, I'll call out here

is the scores. Uh, so we created a few different scores that we'll wanna use to actually score, uh, these prompts. So we have an accuracy, formatting, and completeness score.

And again, this is just in that, that repo, in that resources.ts file. Uh, we have, maybe just to, to point out, uh, linking a little bit of what Carlos was talking about to the actual code that you're seeing.

We have LLM-as-judge scores, as you can see here. Uh, really again, trying to pinpoint here the accuracy, right? Uh, we're not overloading a single LLM-as-a-judge score with accuracy, completeness, and formatting.

Gonna be very, uh, sort of, um, detailed or, you know, scoped down to that particular thing. And then the last one, we have a code-based score, right? So this is a little bit more, uh, binary, right?

Is the, uh, the formatting of this change log that the LLM generated, does it map to what we expect? And so we can use some code to do that.

So that's what we created via that script.

Question.

Yeah.

How do we get that sandbox project into our Braintrust project?

Yeah, so if you go back to the, the lab setup, when you run, when you run that install... So I'm using pnpm.

All right, I don't know if, if you use that. If you're using that locally, you can s- you can also use NPM.

So what is the key for OpenAI API key?

What is the key? Do you not have an OpenAI API key?

No. If... And the instruction says that, uh, reach out to us if you do not have the key. Read it, it says, uh, if in case you don't have API key, please let us know.

Okay. Um, I don't know if we have one to distribute at the moment.

Yeah, if you don't have-

Excuse me. Can I do this later?

Well, if you, if you go, like, the part of the setup here, Reverend, we go to... If you, if you come in here to a playground as an example.

All right, and we're gonna pull in one of those prompts, or we can pull in both of these prompts to do, again, like what Carlos was talking about, that, that sort of like AB testing.

It's gonna ask for some OpenAI models, right? It's g- If you don't configure, uh, an OpenAI API key inside of your Braintrust account-

Oh, sorry

... you don't have a provider to actually run this, this task against.

Oh, okay.

Um, but, but this is, this is the playground, right? This is what Carlos was talking a little bit about earlier, uh, about being able to do some AB testing, right?

I have, uh, my two prompts that I've loaded in. The idea here is to, to load in those different ingredients, right? Our tasks, our, our data set, and then our scores.

So you'll, you'll look down here. We're gonna select that data set. And then we're gonna select the different scores that we wanna score this task against, and I'll load in my accuracy, formatting, and completeness.

Um, can do a couple things here. I can click Run. This will actually, in parallel, go through that, uh, that dataset and use each task that we've defined, and then it will score those, right?

So the idea here is to, again, like this provides that sort of rapid it- rapid iterative, uh, feedback loop that we oftentimes need to build these types of products.

So here are my, you know, like my example rows. Again, these could be synthetic. These could be a, a small subset of rows that are coming back from my application.

But now I can get a sense for prompt A, prompt B, how are these performing with my scores, uh, you know, relative to the things that I have here.

But, uh, then I'm able to do a lot of different things here. I, I can, uh, look at maybe a summary layout, get a sense for, uh, the scores.

So, uh, at the top, this is sort of like my baseline. Up here is my base task, and this is my comparison task. So you can get a very high-level, uh, look at these different scores and how they fared with the different prompts that we've loaded in here.

Now, the other thing that we can do that Carlos mentioned is, uh, experiments. So oftentimes we'll want to capture this-- the, these scores over time, so when we do make changes, we understand how those scores fared a week ago, a month ago, or, or whatever it is.

So I can click this Experiments button, and you'll see the different things that we've loaded up into this playground are already, uh, here within this, uh, modal. We'll click Create, and this will actually create the experiment that you can go to here.

And again, this'll, uh, if I click maybe one out, this is-- allow us to track this over time. This is what we can also lay in, uh, uh, in our CI kind of workflow.

So we go make a change to that prompt and make a change to that model. What are the impacts to, uh, the, the scores relative to what we had over history?

What is the completeness score? Uh-

S-sorry. Can you-

What is the completeness score?

What is the completeness score? Yeah, we can dig into that a little bit. Um,

this is an LLM-as-a-judge score. So the idea, right, we're just gonna give it, um, instructions. The LLM is gonna score the output based on what we've provided in this prompt.

Uh, you'll also note I'm just really pulling in the, uh, the structure of my dataset, right? And so y-you obviously can write... And, and another thing that Carlos mentioned is like scoring the score or eval-evaling the score.

So how well is this thing actually doing, um, based on the output that, that we are seeing within our application?

Okay. That's really activity one, right? It's, uh, reviewing some of that stuff, and then it's creating that playground and showing you all like the, the sort of way that we can, uh, iterate here within Braintrust to create better AI or gen AI products, right?

Like, this allows me to now-- Well, okay, so maybe this isn't the right model. Maybe if I do, uh, maybe I want to see this new GPT model, I can run this, and now I can see how the, the model changed for that particular score, uh, how the scores change when I change the underlying model, right?

But now I have these-- You have like all of these different inputs that could happen to th-these applications. It's a way for us to track and understand when I do go and tweak this thing, there's actual data behind it, right?

This isn't like vibe check. This isn't, um, "Yep, I think that looks good. I looked at this row. It seems like it's better output." This is data behind it, and we can actually understand as we tweak that prompt, tweak that model, how does that impact our, uh, our scoring?

And then again, you can like overlay this within CI and, and so on.

Question.

Yeah.

So, um, I pulled the project down, and, uh, I have the, uh, Braintrust account-

Mm-hmm

... with the OpenAI, uh, key and everything. I just don't, uh, don't know where it should, like, be right now in the project. How do I get the, um, GitHub, uh, project that is on my machine to the-

Yeah. Okay. So you, you cloned the repo.

Yeah. [clears throat] Can I-- And can we get a sense of where people are?

Yeah, please. Uh, g- uh, we can back up considerably. I, I can s-- I can just, you know, with you, we can fix this. Um, you've cloned the repo, correct?

Yeah.

Okay. In your .env.local, do you have a .env.local file?

I should have, yeah.

Uh, there is a .env.local.example file, so you can copy that into the .env.local. Those are the, the keys that we wanna fill in. Have you filled those in?

No, I did not.

Okay. So if you haven't within your Braintrust org created an API key-

Do all the people already did that? I mean, am I the only one?

I'm guessing no.

Yeah, I don't think so. I think the internet connection is probably the biggest thing we're fighting here.

Yeah.

Yeah.

Luckily, all this in-- all the instructions will be available after the workshop, same with the slides. I'm about to share the, the slide link. So tough to update with the connection, but, um,

at least seeing Doug go through the same process will give you an understanding of, you know, what we were hoping you'd have that hands-on experience doing. Um, we don't have too much time to, to wait on this specific activity, so I think in a few minutes we'll keep going, and

hopefully you'll be able to set up your keys so that you can catch up when, when you have some time.

Yeah. Just in the interest of time, we'll, we'll probably move forward. But just to complete that, if you go into your settings within your project, uh, or excuse me, within your, uh, Braintrust org, you'll see API keys.

This is where you're gonna create this API key. You did that. Okay, perfect. Create that. You put that in that .env.local file, and then you run pnpm install.

So that is, that is my-- in my, uh, home directory, right?

That's-- It's wherever you cloned that repo.

Yeah. Okay.

So at the root of that, uh, you put the, the Braintrust API key in your .env.local file.

All right.

And you should have a spot for it already if you're using sort of the template from the example.

And then when you run pnpm install... Again, just to highlight this,

it's actually running this command, Braintrust push. So we're taking the, the resources that we've configured in our, in our project and it's pushing it via our API key to the Braintrust org.

Got it.

Cool. All right. Uh, maybe going forward a, a little bit here to talk about, uh, the, the flip side of this,

right? We've been kind of in the UI for the most part, uh, in our playground doing that, that iteration, changing prompts, changing models, and so on. Uh, I think it's important to understand that we can do a lot of this via the SDK as well.

Uh, we also have a Python SDK, if that's the kind of, uh, flavor you're, you're most, uh, used to using or the, the language. But, uh, that, that top portion here, uh, is essentially what we did, right?

In that, that install or that post-install script. We defined our assets in code. We defined scores. We defined prompts. Um, we defined a dataset even, and then we pushed that into our Braintrust org.

The, the benefit here is that, again, we can use our, our repo, right? We can leverage version control to ensure that the things that we want, uh, to change are actually version controlled alongside of everything else that we're building within that application.

So there are really two modes to actually work with, with the Braintrust platform. It's its UI or its SDK. Again, there's really no limits that we place on, on the, on the user of the platform.

It's gonna cater to maybe a different, uh, persona, a different use ty-- uh, use case. Uh, but you can use kind of both of these different things. The other thing that we haven't done yet, uh, well, that we did, uh, wi-within the playground when we ran that experiment, we ran, we ran those evals, right?

We actually evaluated, uh, the task against the dataset with our scores. You can also define evals in code, right? You can define the evals within your repo, uh, in a very similar command, Braintrust Eval.

Now we can push that up to the Braintrust platform and essentially run that same thing. Track it in that experiments. I have now, uh, an understanding over time how my, uh, my evals are performing as I go and change things, all those different things.

A little bit more, uh, insight here you probably saw with some of that code. This is the, uh, the, the, the command we're gonna run Braintrust push, and you essentially give it either the name of a file that have evals in it, or you can give it the name of a folder, uh, as long as your, uh,

your files have .eval.ts as their naming convention, and we're just gonna go run those evals in that parallel fashion that you saw within the UI.

A couple things maybe, uh, I, I mentioned earlier, but just important to highlight. You should do this when you want source-controlled prompt versioning. You want consistent usage across your, your different environments, and you also wanna leverage online scoring.

Uh, mentioned this, uh, obviously, the, the .eval.ts. That's essentially what the SDK is looking for, uh, when we go and run those evals. It makes it really easy to run a, a larger subset of those without specifying each single file that you wanna go run those evals for.

Um, but you can see, uh, and I'll, I'll-- Let me jump into the actual activity. You can see the eval that we've created. It's, it's what we've been talking about, right?

The three ingredients that we need. We need that task, we need that dataset, and then we need at least, uh, one scorer there as well.

How do you bootstrap a dataset, like with multiple examples? Is there human feedback in loop or is an LLM being used to create the dataset as well?

Yeah. So, uh, the question was like, how do you bootstrap a dataset? Um, I, I think it's a good question. I, I think you could certainly start with syn-synthetic data, or you could even start with, um, you know-- You, you release a feature, right?

You're, you're gonna l-- you're gonna start logging that feature. This is another thing that we haven't yet covered. But you can actually use the logs from that to add to the dataset.

So now you have actual real-life data. The thing to not do is wait until you have a hundred, two hundred, like you have this golden dataset. If you think back to that matrix that Carlos was showing, there are the different ways in which we could start to think about improving the application based on what we observe.

So start with something small, and again, it could be synthetic, but then you can-- Once you start to evaluate it, then you have a different, um, you have different inputs on what you need to do to go and improve an, uh, a scorer or improve the application.

But it's, it's really, I think, maybe up to you. The, the, the, like the best practice or the thing that I would think about is like, don't, don't stop yourself because you only have a small subset of data.

Got it.

Okay. Yeah.

So for the, for the, the tests you're running, if you're using an LLM as a judge, so like for the percent completeness score-

Mm-hmm

... you're using GPT-4.1 as a judge. That's subjectively scoring the test that you're running, right? So like for the, for the sa-- for two runs that are-- that happen one after the other, you could end up with different scores, right?

If you're using like LLM as a judge to run the evaluation.

Um, so the, the question was like, would I get different scores- Um, because I'm using an LLM to do this, right? And it's not really n- deterministic. Um, I, I think that's the, the reason why you would use a better model so you don't see something like that.

Um, I don't know, Carlos, do you have any other thoughts there?

Yeah. I mean, I think you're speaking to the nature of an LLM being non-deterministic. So yes, there may be some variability. What we see our customers do, especially with the SDK, is, uh, do trial evals.

So you will run it maybe five times and then take the average of those. Uh, so there are things that you can do to try to beat that, but it, it is the nature of the beast and you have to learn to work with it.

And then the other thing is, how are you scoring, like, a percent completeness if the task of the LLM is, like, to judge, like, put it in categories like excellent, good.

Are, are you, like, mapping those to, [coughing] to scores or?

Yeah. So the, I, I think the, the question is, and if I come to the score and you look at the completeness, so the LLM here-

Okay.

Yeah.

Got it.

It has to decide-

Yeah

... based, again, on, like, the criteria that we give it, um, if it comes up with excellent, that maps to one and, and so on. But again, this, this score that it gives has to be between zero and one.

Yeah.

Yeah.

It's really helpful when you're using an L- LLM as a judge to go through the Braintrust logs and read the rationale. So it will explain why it chose, you know, 100% or 75%, and you can use that to tune the LLM as a judge and improve it.

Um, it likely will not... You know, you don't want it to say 100% for everything, right? If, if that's the case, you need to improve your evals. Uh, and even if it's saying, you know, 30% for everything, it doesn't necessarily mean that it's performing horribly.

Uh, what really matters is the baseline that you're comparing it to. You know, how did it perform yesterday on these scores, on this data set? Uh, you shouldn't be comparing just, you know...

It needs to be 80%. Not necessarily. It matters what happened yesterday, what you're comparing to, uh, previously.

Yeah. And this is where it becomes really beneficial to, to be able to actually drill into what happened within that task. Uh, so being able to not only understand the, the calls that the LLM makes, the tools that it invokes, but actually drilling into those scores that we're using and then, like Carlos mentioned, what is the rationale

that it gave, uh, to give it a good score here? Does that make sense? This becomes, again, another way of, like, this is the, the human review portion or part of building a GenAI app.

Yeah.

Does Braintree offer, um, any kind of, like, optimization or prompt optimization features? So assuming you have your eval down, you got tons of, uh, different, like, data points to test from.

Um, can you kind of do Braintree to offline the prompts itself?

Yeah, that's a good question, and it's something that we're thinking a lot about is how can we add LLMs to optimize this process for you, not just for prompts, like you mentioned, but also for datasets and for scores.

We're one to two weeks away from releasing our first version of this. Uh, we're gonna call it Loop, and it will do exactly what you're saying. It will help you optimize the prompts and improve your evals.

Just to build on that, imagine, like, this, um, this feature Loop, it has access to previous results as it's doing it, so it understands when it makes that change, is it better than it was previously.

So it's, it's, it's sort of like that agentic type workflow where it has access to tools, but it's able to, uh, to iterate on that prompt and run those experiments, uh, with you, of course, like, in the middle to, to prompt it to do different things.

But it has access to those previous experience, experiments and the results of that, so it knows, like, the general direction it needs to go to get improvements. Yeah.

So how do you build that? [laughs]

We use Braintrust. Yeah, exactly. It's a way of dogfooding. Uh, so h- the question was how do we eval our AI feature, and it's, you know-

Yeah

... of course, we have to use Braintrust. And it's, it's honestly really cool to, to look at the project and see all the logs coming in and, and looking at the scores that we've chosen to go with.

Um, but yeah, Braintrust is really helping, and it was actually something that Ankur, our CEO, has talked about, uh, the process of, of actually getting to a point where we were excited to release something like this.

Previously, the models were not performing to the level that we were needing them to perform to. So every few months, he would run a new benchmark on this specific use case, and it wasn't until, you know, the last month that a model finally, uh, reached that, that expectation.

Yeah.

They gave me a mic, so hopefully you can hear me.

Yep.

Um, cascading off the gentleman in the front's question around, um, the subjectivity of using the LLMs as a judge in these types of cases,

do you offer any way to gain access to that rationale programmatically, such that you could evaluate the thought process of the LLM as it's doing? It's kind of meta, like one-step review-

Yeah

... but adding in that second layer where you could identify weak spots, perhaps if there's something that's hyper workflow oriented or has a very strict process you're looking for the LLM to follow.

Yeah. I, I mean, I, I think you probably saw when I highlighted here, correct me or thumbs up if you saw this or... Yeah. Okay. This is all accessible via, via API.

Uh, coming back down here. So, like, the, the rationale that the LLM gave, [coughing] you could certainly build something o- on top of, uh, the, the, these rationales, uh, and then generate, you know, again, like eval the eval type of workflow.

Cool. Yeah.

You had a question.

Uh, so just curious, like, uh-

There's a mic if you'd like it. Yeah

Yeah. So, uh, yeah. I'm, I'm just curious, like, how should we build our confidence around, like, you know, the result of LLM-as-a-judge because, I mean, uh, you know, how do we trust the eval result?

Because it's after the model, like, evaluate the dataset, and it's could ... Like, we could get, like, you know, uh, maybe a good result, but actually it's ... Like, maybe large model is just overconfidence or something like that.

Is it, like, we need to evaluate our eval results using, like, humans, like, at the beginning so that we can build the confidence? Like, yeah, just, uh, curious any experience, like, you have here.

Yeah, definitely. That's a great question. So, um, I guess everybody heard. Don't need to repeat. Um,

I think there's two things that you can do. One that you mentioned, which is involving a human reviewing that LLM-as-a-judge and confirming that is ... it's thinking the right way, it's outputting the correct score.

Um, another approach that you could do as well as the human in the loop is, uh, using deterministic scores, so full coding functions that are trying to grade the same type of criteria using regular expressions or some other logic.

Uh, that, and you can approximate, right? So if the LLM-as-a-judge is giving a zero, but the, you know, the deterministic code score is giving a way higher score, then you know that there's something that needs attention, needs to be fixed.

The matrix as well that we showed at the beginning that pointed to, you know, should you improve your evals or improve your AI app, that's also a great resource.

Yes.

Thank you. Um, how are you guys thinking about the role, if you think there is a role, um, for traditional machine learning models in evals? I mean, you know, on one hand you have totally deterministic code, and then on the other you have LLM-as-a-judge.

Do you think there's kind of a middle ground for things like intent classification models, entity, uh, recognition, uh, sentiment classification and clustering, and, and kind of more traditional machine learning approaches that kind of sit somewhere in between, you know, the, the, the totally deterministic versus totally non-deterministic spectrum of code versus LLMs?

Like, do, do you think that there's a role for those type of models, and what do you think that looks like?

Yeah, that's a good question. Um, I think it's still to be determined how this all shakes out. There are some, uh, customers, companies that we talk to that are going full deterministic.

They don't use LLM as a judge. And then there are others that are very much going in the LLM-as-a-judge route. And I think the reason that there's a split is 'cause they both work.

Uh, so you know, I don't know. I don't know how this will eventually shake out, if we'll reach in the middle or if, you know, determinism will win. Um, what I can say, though, is that it, it's highly dependent on the use case, the problem that you're solving.

And experiment with both, and then you can determine which one is working best.

I guess those are, like, also largely code-based, right? The, the things that you're talking about, and some of it, they lean a little bit more to- towards that.

Yeah. I, I mean, I'd say there's still kind of parallel approaches-

Mm-hmm

... are still non-deterministically mystic, but thank you.

Yeah. No, I, I gotcha. Um, it's, it's sort of like that, that middle ground. Um,

yeah, I don't, I don't have a great answer for you. Uh, I do think, uh,

using Braintrust, uh, you, you do have the ability to at least configure both of these, the, the LLM as a, as a judge and then the scorer. And then you can, again, using the, the human review process, uh, find the ones that, that actually map to the, the right output the best, and then that's how you start

to build your application. But it's, it's a really good question.

Thank you.

How is the activity going? I know we are getting, you know, last 25 minutes of, of the session. We still have two more, uh, little slide chunks to go through.

So maybe in, you know, two minutes or now we could move to the slides and then, uh, keep it going. Again, this will all be available after, uh, so feel free to keep working on it.

Yeah. Maybe just really quick we could run the, the eval. Just from here you could see, um,

we're actually gonna take the, the eval defined here, right? So we have our, our task, we have our scores, then we have our dataset. Essentially what we just, uh, did within the UI, pushing that into the Braintrust platform.

Uh, and then you can even see, like, this is where it's running right now.

So again, being able to do these i- in a couple different ways, either via the, the SDK or from the, the playground itself.

I, I think you explained this already, and I maybe I was distracted by the Wi-Fi, but how do I think about the difference between the playground and the experiment?

Yeah. That's, that's a great question. Let's see if we can just quickly go back to the slide.

Playground you can think of as quick iteration.

Experiment ... So a playground ephemeral, right? Experiments long-lived, historical analysis.

If that helps answer your question. Um, they're becoming more and more the same. You know, historically the experiments had a bit more bells and whistles, uh, so I ...

You know, typically teams would gravitate towards the experiments. But we found that they really liked the quick iteration. They really liked using the UI. And so we started beefing it up, and now they've become pretty much the same.

So, so yeah. Playground more ephemeral, quick iteration. You want to save the work that you've done to an experiment so that you can later review it and see the scores update and change over, over time.

I guess that was my ... So when I do an eval from the SDK it always is an experiment. Like, what if I just wanna iterate in my-

I, I should use the UI, the Playground UI for that

Yes, and remote evals, which will allow you to define via the SDK, the eval, but then expose it into the Playground. Uh, so some of-- It's, it's like the bonus activity in the document at the very end, so maybe you should check that out and...

W-we won't have time in this session, but if you come to the one at three thirty, we will.

Any other questions? Okay, cool. So moving into lecture three, uh, so this is, you know, once you've finished development, it's reaching customers, you're in production, right? Now what do you do?

Well, the, the important thing is logging. You want some observability. You wanna understand what's going on. How are they using it? Are there any gaps? Are they unhappy?

It can help you debug, uh, a lot faster. It can allow you to measure quality instantly on that live traffic. You can turn those production traces, what you're logging, you can turn it into a dataset and bring it back into the Playground, keep improving the prompt.

And, you know, it, it allows a lot of non-technical people to understand what the end user is thinking. So you can close this feedback loop. We have a lot of PMs, SMEs using Braintrust, and going through the logs and, and looking at that user feedback to understand what gaps and, and improvements may exist.

So how do you log into Braintrust? Uh, well, this is done via the SDK, right? It needs to plug into your production code. Uh, so these are some of the, the steps here, the, the tools that you can use.

So, you know, you ne- you need to an-initialize a logger that will authenticate into Braintrust. It will connect it to a project. So now your logs will go to a specific project in Braintrust.

Um, some great ways to, to get started with really one line of code is to wrap your LLM client. Uh, so you can use wrap_openai, uh, around that LLM client, and now any communication will get logged, uh, with your prompt, response.

It also, uh, metrics, so how many tokens were sent back and forth, the latency, all errors, everything, uh, just by adding that, that wrap_openai. You can do the same with Vercel AI SDK, uh, or you could use Otel.

So we also integrate with Otel, um, so if you want to go that route, it's also available.

If you want to log and trace arbitrary functions, we also support that. You can just use a trace decorator around the function. Um,

really helpful for keeping track of any, uh, functions that are helpful to, to understand and, and keep track of. And then if you need to add additional information like metadata, you can use span.log.

So it's, it's very capable, very flexible, but there's still these, you know, one line code ways to, to get started.

So now that you're pushing all of your logs into Braintrust, you're capturing, you're observing real user traffic, now we're gonna get into that, you know, online scoring piece as opposed to offline, right?

So online is measuring the quality of live traffic. So you can decide how many logs that are coming in will get evaluated and scored. Uh, it could be a hundred percent, it could be one percent, it's up to you.

This allows you to set early regression alerts. So if it starts dropping below, you know, seventy percent, sixty percent, ultimately it depends on the score that you're using and what you've established as the baseline, but if it starts dropping below a critical amount, you can set up alerts and notify the correct team.

Uh, you can also A/B test different prompts. You can, uh, set up tagging and understand, oh, this trace coming from this user, uh, is from prompt A versus this one from prompt B, and you can compare, uh, the, the grades, right, the, the score results coming back in.

So this is great for just improving feedback, moving quickly, and understanding if there's been a drop in quality.

How do you create an online scoring rule? Well, everything is done via the UI. You go to your project configurations, click on Online Scoring, and then you can add your rule.

This is where you'll define what scores you want to be used on that live traffic, and then, uh, crucially, that sampling rate. Uh, so, uh, maybe at the beginning you start with a lower sampling rate, and then you can increase it, uh, once you trust the metrics coming in.

You can also choose what span you want this online score to run on. So it defaults to the root span, but you can get more granular and specify, you know, I want this nested child span to be, uh, scored.

Once you start collecting these logs, collecting these online scores, oftentimes teams want to view them in interesting ways and customize the lenses on those logs. So that's where custom views come in.

Uh, you can apply filters, sorts. You can customize columns on the logs with whatever information you'd like. And now you can start saving those views and making them available for the rest of your team to just come to the logs and select, "Oh, I wanna go to, you know, the logs under a fifty percent view," or, you

know, their own custom view that they've made that's specific to what they care about. Uh, so it's, it's a great way of collaborating and, and, uh, speeding up the process of viewing the important things to you.

Great. So, you know, we went through the slides. Now we would jump back in to the,

to the activity document. There we can look at the actual, uh, code, see where the, the logging is being captured in our files. spin up the application so you can actually view the application in your dev environment, interact with it, and you'll see those prompts and outputs being logged in Braintrust.

Uh, so if, if you've gotten that far and you have the, the dependencies installed, then I would recommend doing a pnpm dev, and now you'll have your application up and running.

Interact with it a few times, and, uh, you'll see that populate in your project logs.

Once you do that, then you can go to your online scoring settings, set up a rule, and you can keep interacting with the app, and now you'll see it, um, populate with that online score that you just enabled.

Maybe a quick example for, for those that are still having sort of Wi-Fi issues. Uh, come back down here.

So I'm gonna come and, uh, spin this up. So as Carlos mentioned, uh, pnpm dev. This is gonna, uh, spin up that server on [REDACTED:url]. Uh, you should see

something that looks like this. Uh, there's a few things that you can just click these. This is sort of like the easy button to get going. Uh, this is the, the GitHub URL.

It's, again, it's looking for the, the commits that, uh, have been made since the last release, and it's gonna summarize, uh, again, using the, the prompts that we've configured here, and then start to categorize them.

But now the, the, the interesting part of this, if you come back into the Braintrust platform,

is if you look at the logs. So this is what just happened on the Braintrust side. Uh, so we sort of have this top level, uh, trace, the generate change log request, and then essentially the, the, the tool calls you can think of as, right?

We're getting the commits, we're getting the latest release, we're fetching those commits, we're then loading that prompt. So we're actually loading the prompt from, from Braintrust, and then you can start to...

You know, you can click through a lot of these, and as Carlos mentioned, when you use those wrappers, you get all of this sort of goodness out of the box, right?

So what are the, the number of prompt tokens? What is the estimated cost? This is, uh, this becomes really helpful as you start to monitor over time, right? Uh, probably not set up yet because we don't have, uh, too much going on, but, like, actually understanding what does that token, uh, amount look like over time?

What does the cost look like over time as we change models, uh, and so on. But that's how really easy this is, and maybe just to complete that, that loop, if you come over to the, uh, resour- not the resources, the app generate route.ts, you'll start to see some of this stuff.

So I'll just highlight a c- a couple things. We're wrapping this SDK, uh, AI SDK model. So this, again, is, is how we're really getting all of that, that, that metrics and, uh, it's, it's really allowing for us to log a lot of that information with very little lift from, from ourselves as developers.

But then you also have the ability to, uh, configure things in a different way. So maybe we have, um, different inputs or different outputs that we want to actually log in a particular span, or actually we wanna log metadata, right?

This becomes really powerful when we wanna actually go into those views, right? And we can actually start to filter these things down. We can even filter by that metadata.

So, uh, this is where, again, you can y- hit the easy button. We're gonna wrap our, our LLM client with those SDKs, or we can actually get a little bit more detailed and start to log, uh, the, the particular input and output information that we want, metadata, and now we can sort of, uh, set these different things.

So if I come out here, uh, I can even create, uh... Actually, when we lo- when we add in the scores here, we can create filters based on those scores.

So I wanna create a view that says, "Hey, anytime my completeness score goes below 50%, I wanna create a view for this." This is gonna enable my human reviewers to go in and actually understand that.

And then if you look up here in the top right, we can actually add this span to a dataset really easily. So we find this thing, right, that'll actually add a lot of value in that offline eval ki- type of process.

Click this. Now we have a net new row in that dataset that now adds a lot of value, right? This is sort of like that, that feedback loop, right?

We've, we've done that offline eval type of work. We have, uh, found the right prompt, the right model, all these different things. It's in production. We are logging it.

Now we're sort of, um, understanding that in maybe in a human review type of way. Add that span to the dataset. This adds, again, to the, the offline type of portion.

Again, you just see this, like, this sort of flywheel effect of creating really powerful gen AI apps.

Yes.

Is there an eval score, uh, generated for this online log as well? Like, as we run it, is there an eval score created for it? And do we add it to the dataset based on if the eval score is good because we don't want, like, bad examples in the dataset?

That's one way of thinking it. You don't need to run an eval as the user's interacting with it. That's what the online scoring does. So once you set up the on- online scoring rule, it'll output a score based on the judge that you've chosen as the, the online scoring rule.

Okay.

Um, and exactly what you said, right? You could either filter it and select the good responses, add those to a dataset, or vice versa, s- select the bad responses and understand why are they bad, how do I improve them, make them better.

Just to complete that and, and kind of, uh, how do we configure this, right? We, we-- Carlos walked through, like, what this would look like. Um, this is, you know, my sc- my rule, right?

Obviously, you would call that something a little bit better, but I actually wanna, you know, uh, add in my scores for those online logs, right? Then you would-- You probably wouldn't do 100%, but we're just gonna do that for, for this instance.

And now when I come back to, uh, my application here,

right, maybe I wanna do, uh... I'm gonna just do a quick refresh.

So now when these logs, uh, happen within the Braintrust side, we're actually gonna run those scores against that output. So we'll understand, based on what happened here, how did it score on formatting?

How did it score on correctness, um- And then this also, uh, can now layer into-- So you can see here, right, we have a twenty-five percent on the accuracy, on-- a hundred percent on the, um, the completeness.

So maybe we have a little bit work to do. But now if I click into this, this is where you can start to now create different things within, uh, the, the, like, the, the view portion here to ensure, like-- So this is a filter.

So maybe I wanna change this to anything that is less than, let's say, fifty percent.

Now I can save this as a view, and my human reviewers are able to now come in here, open up this view, and look at for, look, look at all of the logs where my, uh, accuracy score is less than fifty percent, and now we can again create that sort of, uh, iterative feedback loop.

Any questions on this section? Yeah, maybe a good segue to, uh, the, the human in the loop. Um, this, this becomes really, uh, y-you almost, you almost really...

Oh, sorry.

Um, I had a question about using Braintrust and implementing it on existing projects. Is it, is it something that's easy to do with, like, a, um,

something like LangSmith, you can just add, like, a couple lines, and it'll trace everything for you? Is it the same in Braintrust as it, or do you have to refactor all your prompts to use, like, the Braintrust prompts?

No, it's, it's essentially the same thing. So, like, you have-- LangSmith has code to wrap, uh, an LLM client, right? Or it has, uh, decorators to, uh, put on functions.

Mm-hmm.

It's the exact same thing on the, on the Braintrust side.

Got it.

Yeah.

So then if you do that, is it easy to then use, um, like, create datasets from those logs?

Yeah, absolutely. As long as you're, uh... th-the, the logs that you're producing map to the structure of the dataset that you've created, then absolutely. It becomes really just you click that button, we're gonna add that span to the dataset, and it becomes really easy to connect those two things.

Mm-hmm. Cool. Thanks.

Yeah. Yeah, of course.

Yeah, good question.

Um, okay, yeah. So let's talk maybe a little bit really quickly about the, the human-

Oh, there's a question over here.

Oh.

Or no.

Super, super quick. Uh, with the sampling rate, is there a way to just override that, where if certain inputs are received, you can force that to be included in your sample set?

Like, if you have some manual user input, you know, just push that back into the system automatically if it's not hitting your sample rate.

The way that you would go about that is changing the span that is targeted. So instead of it applying to the root span, you would specify a span that only happens if a certain criteria is met, right?

Okay.

So y- then it could be a hundred percent or fifty percent of just when that span appears. Yeah.

Okay, so yeah. This is where, uh, we could, we could, uh, bring sort of, like, the human-in-the-loop, uh, type of workflow. This is where we wanna actually maybe bring in subject matter experts.

And, and Carlos mentioned, like, uh, product managers, maybe SMEs. We also have doctors coming into the platform and actually evaluating some of this stuff, right? These are the people that actually understand whether or not that output that's created by that large language model is valid, is good, right?

And this is, uh, a really powerful thing to have as part of the process to, to building really powerful AI applications. Uh, we can catch hallucinations. Uh, being able to establish that solid foundation or that ground truth is oftentimes, um, you know, having that human-in-the-review, uh, in-the-loop type of person becomes really, uh, beneficial here.

Uh, so why does this mattor- matter? Excuse me. Uh, it's, it's really critical for quality and reliability. Uh, like we were just talking about, like, with LLMs and being able to trust whether or not they can do the same thing over and over again.

It's non-deterministic. Uh, automations can miss that nuance, right? We wanna be able to sort of apply that, that human type of, um, review to the, the, the things that we're doing on the AI side with LLM, LLM as a judge type scorer.

Uh, we also want to, um, help you make sure the final product meets the, the actual expectations of the user, right? So, uh, th-the user is gonna have a much better understanding of how to...

or, like, what the, uh, final output should be. So having that person in the loop to look at those outputs, um, becomes really powerful to ensuring that you build really, uh, really strong gen AI applications.

Uh, two types of human-in-the-loop. Uh, there is the, uh, the human review. Uh, so this is where these, uh, these people are actually gonna go into the Braintrust platform and actually manually label, score, and audit those interactions that the, uh, that the user had with the, the AI application.

And then there's actual feedback from the user real-time. Uh, so this is, like, you know, a thumbs up, thumbs down button within the application saying, "Hey, you did a really good job," "You did a really bad job."

But now we can-- Y-you can, like, sort of use these together as well. I can now create a, a view within Braintrust that filters down to any of my user feedbacks of zero, so thumbs down, and actually wanna review whether or not those things are bad.

And if they are bad, I can add those to the dataset, again, creating that, that sort of flywheel effect.

Um, just really quick here. If, if you go-- If you look at that application, you're able to actually, uh, click one of these, you know, thumbs up or thumbs down, uh, create a comment, "Really good," and then this is now logged back to the Braintrust application.

So if you look back at our logs,

I'll remove our filter. Uh, so we should have a user feedback score now here of a hundred percent, and then we should have a comment over here as well, "Really good."

But then again, these are the things that we can now, if I open this back up, I can create a filter on my user feedback score. Uh, now I wanna understand all of my, uh, my logs where user feedback is one or zero, and then I can do something from there.

But this is, this is done very easily via the log feedback function within, within Braintrust. You provide it sort of, like, the, the span that, that was already created within that log, and then you just, you log, uh, you, you provide that user feedback to it.

Um, you, you can also enter, uh, really quickly here in the platform, you can enter, uh, human review mode. So this is a way in which, um, i- it's sort of hiding away some of the, the different, uh, you know, some fields that may not be really relevant for those people that are coming in and doing human

review. Oh, I haven't actually configured any scores yet, so you can actually see, uh, you would come out here, and you would create different, uh, scores for that human to go in and do that review, whether it's, uh, sort of like an option-based, uh, free form input, uh, slider.

So is this, you know, maybe thumbs up, thumbs down, sort of like yes or no, or maybe you could do something a little bit more verbose, um, A, B, C, D, whatever it is.

But this is where you, you create these scores. Now they exist as part of those logs. Those humans can go in and now look at the input and the output, give their review of it.

Again, adding to, uh, that, that again, like that, that flywheel effect that we need to create.

Yeah. I wanted to add there as well, this is really helpful for eval-ing your LLM as a judge. Oftentimes, we see customers use this process to provide some ground truth for their datasets and also for the LLM as a judge, right?

So you can imagine having a team of SMEs come in, they review, uh, they do thumbs up, thumbs down on maybe five different, uh, criteria qualities that they're measuring, and then they provide that data to, uh, a playground where the prompt is the LLM as a judge, and they go through the playground, and they test to make

sure that the LLM as a judge prompt matches what the humans thought. So just something there to think about. Uh, but as Doug is saying, it's a great feedback loop.

It's a great, uh, flywheel effect that can be created when you add this human to, to verify and confirm.

Cool. That, that is it for, uh, the, the workshop. We, we do have a few minutes left. Could certainly answer a couple more questions.

So, uh, for people who are successful with this, how much time are they spending going backwards and forwards, putting humans in and validating the tests themselves before they get into like more live testing?

And then how often are they going back? What's the kind of balance of like time on task?

Offline versus online evals, like...

Yeah. And how much, and how much you have to do upfront to really get the best results, or can you really just put something down, figure it out later and optimize on the fly?

Um, you know, 'cause you don't wanna get stuck in analysis mode for just a bunch of-

Right. Yeah, the question, just to repeat it, is how much time do you have to invest upfront to get value? Should you keep going over and over it to try to optimize or better to just start quickly with minimal scores, minimal dataset, and then, uh, keep improving?

And I would say the latter, right? You don't want to, to be fixated on creating a golden set-- dataset or 20 scores. Like if you have one or two scores and you have 10 rows in a dataset, it's going to be tremendously helpful.

And then from there, it's all about iteration. So just going back and improving, adding some more rows, adding another score, tweaking the scores. But you really just wanna get started quickly.

Yeah.

Um, so, um, you've mentioned, um, some elements, uh, of this, um, uh, scoring. Uh, that's the, the function that you want to test, that you have to define the, the test steps, if you will.

Um, one of the challenges that we are finding is our actual application does change and, uh, it could change bi-weekly. It could change monthly. Is there a way to look at, um, uh, trying to automate changing the actual function that you now need to change to match the way that your application logic has just

changed, uh, this week from two weeks ago?

Would you say-

So-

Oh, go for it.

Yeah, I guess I was just gonna, again, uh, clarify to, to make sure I understood. So you're saying that the scorer, the, the actual scoring function is going to,

uh, stop being useful. It's gonna become obsolete. It's gonna become too old to actually gauge the quality.

Not just the scoring function, but the actual steps that you want to, uh, test. So, you know, this week there might be only two turns, just giving a very simple example, and in two weeks in the next sprint, there are now five turns in your app because the logic has changed and now you, you have to update,

of course, the, the, I think the function element. There's probably no way around it. I'm just curious about, uh, whether, uh, you guys have, uh, thoughts about how that could be improved or, or made easier.

Well, I think your, your task al- will, will always change, right? Right. The thing that we're trying to, to build-

Yes

... that's where Braintrust helps because we're gonna understand when we do go make that change.

Yes.

We actually understand whether or not that change improved our application or regressed it. So like we're not gonna say stop making changes to the underlying-

Yes. Yes. No, I'm-- I, I real- I understand that. So it, it is inevitable that the, the application is going to be, uh, changing.

Yeah. I-

And you're gonna have to constantly update the current-- the, the test, uh, the test that-- the function that you're actually wanting to mimic in your test.

It's very similar to traditional software testing.

Yeah.

You don't wanna write a test that lasts for a day or, you know, a week, right? You want to think of, of robust, uh, tests that will live on for months or years and will actually measure the underlying quality of the application that will be long-lived.

So the-- I, I think it's more of how do you optimize the scores to measure qualities that will still be around, even if you add some additional steps in the task.

... than that because unless you have those additional steps in your function, you're not mimicking your, your application's, uh, logic. Y- you're still using the logic from two, you know, last sprint.

So no matter how good your scoring could be, it's not, no- it's no longer reflecting what your application is doing this week-

Well, I, I think-

... or, or this sprint.

I, I think like regardless of like how many steps you have, like there's still an input and there's, there's still an output that we wanna score against.

Correct?

Uh, y- yes, but, um, I think one of the things you need to do is to first define how you're going to arrive at the score. The, the input comes in, and now maybe you have three turns, and then because you're mimicking your app, and then you get your output from these three turns.

Your app just got upgraded. There are now seven or five turns or whatever.

Yeah. So the ... When you're writing the evals, you can dynamically call the task. So even as you're working on your application and it's changing-

Mm

... you're still pointing to the, the changing app. So the idea is that when you are wanting to merge into main, you open a PR, and then your evals will run on those new changes.

You don't need to go in and update the .eval.ts files. They will now reference the updated, uh, task application that you're trying to understand the underlying quality for, if that makes sense.

So I think the, the question again is, are the scores, is the underlying logic something that you can trust [laughs] and that will live on? Um, again, it's not easy, and it's something that is changing, uh, but that's, that's what we're hearing from customers is investing in that.

Uh, at Braintrust, when you send eval to humans, SMEs, what's the name of that role, and how are you managing that? Like, I'm guessing to some extent it was originally the team, right?

But that can't scale, so how are you managing that?

I think it's like organization specific. I don't know if there's a, a specific-

I'm saying your organization. Using your own tool, how are you managing the SMEs yourself?

Hmm. Hmm. I don't think we're using any SMEs at the moment. We don't... We're not a healthcare company or a legal tech company where we heavily rely on specialized knowledge in that degree, you know?

Um-

But you're not doing human evals of your own product?

We just now started, we just now branched into having an AI component to our application. Uh, so we haven't needed to, to go there just yet, but we, you know, we talk to a lot of customers that are working in, in those specific industries with those use cases, and they will sometimes hire external, uh, services that will

do the, the, the annotations for them, or they'll bring them into Braintrust and, you know, they'll be using the platform just to, to review, so they have a specific role within Braintrust, and there's a specific view that they would operate in that's just for annotation.

Got it.

Yeah. Great, yeah. Another question over here.

Uh, I just, I was just curious that because we're using out-of-the-box AI models here and are not really fine-tuning the model as the application progresses, are we, do we have a way to like do some few short example prompting from the dataset and the eval scores that we are already using?

So is there some feature like that where I can use the datasets or the online logs that are added to the datasets? If the eval score is good, use it as an example for future prompts to just make the prompt better because the models are out of the box.

Yeah. So question around few shot prompting, providing examples to the prompt of the ideal response. That's something that you can do today with in the dataset. In the metadata column is where you can provide the, the few shot examples that you want for each row, and then when you're running that eval or messing around in the playground,

it'll reference the few shots in the metadata.

Got it. But what about like the online testing stuff, right? Or the online logs, whatever you call it. Like when users are actually using the application and it's hitting the prompt, there can the prompt real time use those, uh, examples from the datasets as well?

It... Right now it's not, it's not, not something that Braintrust facilitates. Within the SDK and building your own logic, like you could come up with a workflow like this, but natively in the platform, we're not facilitating like live traffic into few shot examples.

Got it. Makes sense. Sounds good.

Great. Well, thanks everyone. I know we're over time. Really great to have you all here for our first workshop of the day. I hope you can walk away with some ideas of how you can improve your eval workflow and, you know, our team is here.

We have a booth, uh, just outside of this, so feel free to stop by. We can answer more questions, have a conversation. Yeah. Thanks everyone. Thank you all. [outro music]