Evals 101 — Doug Guthrie, Braintrust

Talk transcript

1. 0:00

   [upbeat music] Hey, everybody.

2. 0:17

   My name is Doug Guthrie. Uh, I'm a solutions engineer at Braintrust. Um, as you can see here, we're an end-to-end developer platform for building AI products. We do evals.

3. 0:28

   If you, if you watched the, the keynote this morning, you saw our founding engineer jumping up and down on stage yelling, "Evals." I am not gonna do that. I'm not as, uh, as funny or cool as him, but, uh, we should all be very excited about evals here.

4. 0:43

   Very brief agenda of, of what we'll cover today in this sort of, uh, this intro. Uh, very, very brief company overview. Uh, give you an intro to evals. Why you-- why would you even, uh, start thinking about using them?

5. 0:56

   What are they? Uh, what are the different components that you need to create an eval? Um, some more Braintrust-specific things via, uh, running evals via our SDK. You'll see, uh, in the examples, um, that you can run evals both, uh, in the platform itself as well as the SDK.

6. 1:14

   I think it's, it's a really kind of cool thing that we can connect, uh, maybe the local development that we're, that we're doing with what we're doing within the platform.

7. 1:23

   Uh, and then how we then move to production. Uh, this is, uh, maybe human review. This is online scoring. So how well is our, our application performing, uh, in production?

8. 1:33

   And then lastly, a little bit of human-in-the-loop, uh, getting user feedback from your users. How do we now, uh, take some of these, uh, these production logs and feed them into the datasets that we're using in our evals, uh, creating this really great flywheel effect?

9. 1:51

   Cool. Quick, quick company overview. Um, you see up there, maybe in the top right, uh, some of our fu-- uh, excuse me, some of our investors. Uh, maybe a quick call-out on the, on the leadership side.

10. 2:03

    Ankur Goyal is our CEO. Uh, maybe the, the reason to call out that is, uh, Ankur in the last, uh, two, two stops, he, he essentially built Brain-- uh, Braintrust, uh, from scratch the last two places and, and this is really where he found the, the idea to like, like maybe this is actually a, a thing that,

11. 2:22

    that people need and, and really like the origination story of Braintrust. Uh, the other thing to call out here is like we already have a lot of companies using Braintrust in production today.

12. 2:34

    This is just a, a few of, uh, the, the companies that are, that are utilizing us for, for evals, for, uh, for observability of their, their gen AI applications.

13. 2:45

    So I won't bore you too much with, with that, but, uh, let's jump into this. If you're at the keynote, you probably saw a, a similar slide here. I didn't, I didn't take it out, but, uh, uh, you see like the, the tech luminaries here, I think as, as Manu referenced them, talking about evals and the importance

14. 3:00

    of them. I think this is a, you know, obviously this is an intro to evals, uh, track, and what better way to start this out with, uh, some, some really, um, you know, influential people in the space talking about evals and, and why they are so important.

15. 3:15

    So why would you think about evals, right? They, they, they help you answer questions. So here's, here's a few of them. Um, when I change the underlying model to my application, is it, is it getting better or is it getting worse?

16. 3:29

    When I change my prompt, right, or when I change, uh, certain, certain things about the application, is it getting better or is it getting worse? We wanna get away from, uh, not having a, a rigorous sort of process around building with large language models which, as you all know, right, the non-deterministic outputs, uh, creates somewhat of a

17. 3:47

    challenge. And without, uh, evals in place, becomes really, really hard to, uh, to create a good application that we can put into production. Maybe some other ones, like obviously being able to, uh, uh, detect regressions within the, within the code.

18. 4:02

    I think the other, the other thing that Ankur mentioned, uh, to me when I first started, which I didn't really mention this, but, uh, this is my, my third week at, at Braintrust as a solutions engineer.

19. 4:10

    But one of the things that he mentioned to me that I thought re-really resonated was, uh, evals are a really great way-- I think people think of them as almost like unit tests for, uh, for, you know, our applications.

20. 4:22

    But he kinda described it another way of like this is a really great way for us to play offense, uh, as opposed to just playing defense, where I think maybe unit tests are, are, are kind of used for.

21. 4:31

    This can actually be used as a tool to, to really help, uh, create a lot of rigor around us building and developing these applications and ensuring that we actually build, uh, things that we can put into production.

22. 4:44

    Maybe from like a business perspective, uh, why would you think about running evals or using evals? Uh, here's a few here. Uh, if you have, um, evals running both offline and online, you create the-- this feedback loop or this flywheel effect I think Manu mentioned in the, in the keyno- the keynote, excuse me.

23. 5:03

    Um, and this flywheel effect allows us to, you know, uh, cut dev time, allows us to enhance the quality of this application that we're putting out into production. If you're able to connect the things that are happening in, in real life with your users in production and being able to, uh, filter those logs down, add those spa-

24. 5:21

    spans to datasets, inform what you're doing in an offline way, creates, uh, that, that flywheel effect that becomes really powerful from a development perspective.

25. 5:32

    Uh, again, a little bit more on the, on the customer side. Here's a few of our customers and some of the outcomes that they've, that they've seen using Braintrust, whether it's, uh, moving a little bit faster, pushing more AI features into production, or just increasing the quality of, of the applications that they have.

26. 5:48

    Here's a few of the, the outcomes that we've seen or that they have seen.

27. 5:52

    So let's, let's start talking a little bit about the, the core concepts of, of Braintrust. Obviously, we're here to talk about evals. Uh- The, the, the things that I think, um, you see like these arrow, these arrows going, uh, one way and, and the other, this again is that, that flywheel effect that, that I described earlier.

28. 6:11

    Um, there's the, the prompt engineering aspect of this. Uh, in Braintrust, think of, uh, th- this playground that we have as an IDE for LLM outputs. Uh, the playgrounds allow for that rapid prototyping.

29. 6:23

    As we make those changes, as we change the underlying model, what is the impact to that, to that, uh, particular, uh, task or that application? Then those evals allow us to understand the improvement or the regression of those changes, and then the observability aspect, right?

30. 6:37

    This is the, the logs that we're generating in production, the, the ability to, uh, uh, have a human review those logs in a really easy, uh, intuitive interface, and then have user feedback from actual users be logged into the application as well.

31. 6:52

    So, what is an eval? Yeah, you probably, uh, heard this, uh, several times y- today and throughout the week if you stopped by the booth. But sort of our definition here is that structured test that checks how well your AI system performs, right?

32. 7:04

    It helps you measure these things that are important, quality, reliability, uh, correctness.

33. 7:13

    So, what are the ingredients in an eval? Right? I've been talking a little bit about task, right? This is the thing, uh, the code or the prompt that we want to, uh, evaluate.

34. 7:21

    The, the really cool thing about Braintrust is that this can be as simple as a, as a sim-- uh, excuse me, a single prompt, or it could be this, this full sort of a- agentic workflow where we're, uh, calling out to tools.

35. 7:32

    There's no sort of limit onto the, uh, the, the complexity that we put into this task. The only thing it requires is an input and an output. The second thing is a dataset.

36. 7:41

    This is our, uh, real world examples. This is, uh, essentially what we're gonna run the task against to understand how well, uh, our application is performing. And how we do that is via scorers.

37. 7:53

    So, the scorer is really the, the logic behind your evals. There's, there's a couple different ways to think about this. There's the LLM-as-a-judge type scorer. So, uh, you give it the, the output and some criteria, and it is able to assess, uh, like say, "I want, uh, based on this output, is this excellent?

38. 8:11

    Is this fair? Is this poor?" And then those outputs then correspond to, you know, zero, point five, or one. Uh, you also have code-based scorers, right? These are maybe a little bit more heuristic or binary, but, uh, we can use both of these to really aid in the development of that, of that eval and ensuring that we're

39. 8:28

    building a really good application. Uh, I think I just sort of mentioned this here as well, but like the, the two mental models here of, of evals, there's, there's offline and online.

40. 8:40

    Offline is pre-production. This is us actually doing that, that iteration. It's, uh, identifying and resolving issues before deployment. Uh, this is where we're defining those tasks. It's where we're defining those scorers.

41. 8:53

    Uh, online evals, this is that real-time tracing of, of the application in production. It's logging the model inputs and the outputs, uh, the intermediate steps, the tool calls, everything that's happening.

42. 9:04

    It allows us to diagnose performance and reliability issues, latency. Uh, based on how you instrument your, your application with Braintrust, we can pull back lots of different metrics related to cost and tokens and duration, and all of these things be-- help inform, uh, how we build this application.

43. 9:23

    Um, I'll, I'll jump into a little bit more of like how we can instrument our app, uh, for online evals. Uh, we're gonna first, I think, talk a little bit more of the offline.

44. 9:33

    Before doing that, uh, maybe just like level set on how to improve. I, I think one of the, the, the, the things that I've seen, uh, in the last few days here, the conversations that I've had, it's like almost how do I get started, or what do I do if X, or, you know, those types of questions.

45. 9:51

    Another thing I heard from Ankur, uh, very early on is that like just get started. Create that baseline that you can then iterate and build from. Uh, I think a lot of people get caught in like creating this, this golden dataset of test cases, uh, that they, they can then like iterate from.

46. 10:07

    Start-- You, you don't necessarily have to do that. Start and build that, uh, that baseline. Establish that foundation that you can then improve upon. But this is a really good sort of matrix of like, uh, if I have good output but a low score, what do I do, right?

47. 10:21

    Improve your evals. If I have bad output and a high score, improve your, your evals or your scoring. But really good kind of like high level, uh, understanding of, of where to start to target your, your efforts when you are building these apps and you're, and you're creating evals.

48. 10:39

    So, let's jump into the actual components that I just, that I just talked about. So, within the Braintrust platform, we have a task. Uh, again, this could be a prompt.

49. 10:47

    This could be like this full agentic workflow. Uh, very basic. You, you see that, that GIF running. This is a prompt within the platform. You, uh, specify an underlying model that you want it to use.

50. 10:57

    You give it a, a system prompt. You can also, uh, give it access to tools. It has access to musta- mustache templating, so you can pass in variables like user questions or, um, the, you know, the input from the user, its chat history, or metadata, right?

51. 11:11

    So, when we actually go and wanna parse through these logs, the metadata becomes actually beneficial, uh, enabling us to do that in a really easy way. Um, going forward, uh, maybe we have a multi-turn sort of chat type scenario where we wanna add, uh, additional messages for the system and the assistant and the user, uh, and our

52. 11:31

    tool calls as well. The, uh, the, the platform allows for that just via this, uh, plus-- this messages, uh, button, and then you're able to add those different messages to the prompt.

53. 11:43

    Um, also, we can add tools. So, oftentimes, the, the, the prompt will, will need access to, to something, right? Maybe it's a, it's a RAG-type workflow. Maybe it's doing web search.

54. 11:52

    Whatever it is, we can now use those tools a- as part of that prompt, right? And so when you, you sort of encode in that prompt, like make sure you use X tool, uh, this prompt has access to that tool while it's running.

55. 12:07

    Uh, the last one, this is actually a feature that is in beta right now. Uh, it's actually creating more of that agentic-type workflow within your, uh, within the Braintrust platform itself.

56. 12:17

    So it's, it's a way right now at least to chain together prompts where the, the output of one now becomes the input of the other. But if you think back maybe to, to this slide, if I have sort of, um, this prompt that has access to tools, you, you create a pretty powerful system here where you're able

57. 12:33

    to go from, uh, maybe that first step that has acc-access to a certain tool, and we get some output from that. We can then go to that next step that has access to maybe some other tools, right?

58. 12:44

    This sort of like maybe multi-agent type of workflow we can create with the underlying tools within those prompts.

59. 12:52

    The second thing that I talked about is datasets, right? These are our test cases that we want to give to the, the task to run, so we can sort of iterate over that.

60. 13:01

    We can get the output and then going down a little bit further, actually score that. But this is, um, obviously really important when we're, when we're running our evals and then when we are, uh, trying to pull from production, right, the, the actual logs that are happening, we can add those, those, uh, spans, those traces to the

61. 13:18

    datasets in a really easy way, and I can show you what that looks like. But, uh, if you look there at the bottom, the only thing that's required is the, the input.

62. 13:25

    Uh, you also have the ability to add, uh, expected. So what is the expected output for that input, right? You can sort of like, uh, create some sort of scorer that looks at the output with the expected.

63. 13:36

    There's a, a scorer called Levenshtein that allows you to, you know, measure the, the, the difference between those two. So you can do some different things based on what you provide to that dataset.

64. 13:45

    You also have metadata as well, again, being able to filter down different things, pulling the dataset maybe into your own code base, and I want to filter by, again, X, Y, or Z via the metadata.

65. 13:54

    That's all possible. Uh, I mentioned this a little bit ago, but just start small and iterate, right? You don't have to create this golden dataset to, to get started here.

66. 14:04

    Um, just re- just start and then, uh, then continue to iterate and build from that baseline. Uh, the human review portion also becomes really powerful. Again, when we have stuff, uh, being logged within production, having humans actually go through those logs, a-and, you know, there's lots of different ways to filter it down to the things that they

67. 14:23

    should be looking at, and then we can now decide to add those things to certain datasets that then inform, uh, the offline evals that we're running.

68. 14:34

    Uh, the last thing, the last ingredient here that we need for our, [clears throat] excuse me, uh, for our evals are our scores. We have both code-based scores, right? Again, this is like more of those binary-type conditions.

69. 14:46

    But you can actually, uh, code TypeScript or Python. You can do that within the UI, as you see over there on the bottom left, or you can, within your own code base, create that scorer and then push it into Braintrust, so we can use it in the platform.

70. 14:58

    Other users who maybe aren't in the code base can use that scorer as well. The other scorer that we have access to is called LLM-as-a-Judge. So this allows us to use an LLM to sort of judge the output.

71. 15:10

    We can give it the, the set of criteria that indicates what a good or a fair or a bad score or whatever it is. You, you get to decide, uh, what that looks like.

72. 15:19

    So you give it that criteria, and it says, "If it's good, uh, I wanna do a one. If it's bad, I wanna do a zero." But this starts to create the scores, uh, that we can use in that offline and that online sense.

73. 15:31

    The other thing to call out here is that we have, um, internally built a package called AutoEvals. So this is something that you can now pull into your project.

74. 15:40

    These are out-of-the-box scorers that are both LLM-as-a-Judge as well as code base. And so it just allows you to get started very, very quickly. Another thing I heard Ankur mention is, um, maybe starting with Levenshtein, maybe not the best scorer in, in a lot of cases, but again, it establishes a baseline.

75. 15:57

    Very, very little development work for, for, uh, our users, but it creates that thing that we can then build from. And now you have, uh, a direction, a direction to go in to go build maybe that more custom score.

76. 16:11

    Some of the things that we've heard from our customers, uh, some tips that, um, that, you know, important to think about.

77. 16:18

    A lot of our customers are using higher quality models for scoring, uh, even if the prompt uses, uh, a cheaper model. Just, just makes a lot of sense, like, while we're running that application to use the cheaper model but use the, the more expensive one to actually go out and score it.

78. 16:32

    Um, also break your scoring into, uh, very focused areas. So, uh, the example that I'll show is a, an application that, that generates a change log from a series of commits.

79. 16:43

    So I could create a scorer that says, "Assess my accuracy, my formatting, and my correctness." Uh, or I could create three different scorers that assess accuracy and then formatting and then correctness.

80. 16:54

    So have your scorers be very targeted to the thing, uh, that they're supposed to be doing.

81. 17:00

    Uh, test your scorer pom-- excuse me, prompts in the playground before use, and then avoi-avoid overloading the scorer or prompt with context. Uh, focus it on the relevant input and the output.

82. 17:14

    A couple things here. Here's where, uh, over on the left we have our playgrounds. This is where we do that, that sort of like rapid iteration where we can pull in those prompts, we can pull in those agents, add our datasets, and add our scorers, and we can click Run, and it will go out and sort of

83. 17:28

    churn through that dataset that we've defined and will give you a sense for how well your task is performing against the dataset with the scorers that we defined. But this is the place where, uh, developers, uh, PMs, uh, we even have, uh, a healthcare company that has doctors coming into the platform and interacting with the playground and

84. 17:45

    even doing human review as well. Uh, depends a little bit obviously on, on the organization. The thing on the right is our experiments. This is our sort of like snapshot in time of those evals.

85. 17:56

    So imagine now, like, as we are doing this development and we're trying to understand, uh, like the last, you know, the last month or so, are, are we getting better, right?

86. 18:04

    The, the, the changes that we are making, the model changes, whatever it is, are we improving our, our application? And the experiments is a really great way to, to understand that

87. 18:15

    Uh, really important maybe to call out as well, you can see in the bottom right, the evals can happen from, uh, the application, right, the, the Braintrust platform, as well as via the SDK. [clears throat]

88. 18:26

    Cool. Maybe just really quick because, uh, nobody likes looking at, at slides all the time. I, I certainly don't. Uh, maybe if you haven't seen Braintrust yet, th-this is maybe a good, a good quick demo.

89. 18:37

    Uh, so again, like the, the, uh, the idea here is I have this, this application. I'll just give you-- I'll show you over here. Uh, you give it a GitHub repository URL.

90. 18:47

    It, uh, grabs the most recent commits and then creates a change log from there. And then once this completes, you can even provide some user feedback. But this is the thing that we want to, uh, evaluate.

91. 18:59

    So what I can do, uh, I'll go into my, my playground. Right? This is the place where I can start to, uh, run those, those, uh, those experiments, or I can start to iterate on that prompt that I have.

92. 19:10

    From my project, I've actually loaded in two different, uh, two different prompts. So maybe I'll-- Before going into the playground, I've actually created these two prompts within my code base, and I've pushed them into the Braintrust platform.

93. 19:23

    I've also created a dataset in that code base, and I've also created some scorers. Right? These are the ingredients that we need to run our evals.

94. 19:32

    So now when we have, we have those, we have those different components, now we're able to start to iterate here. So I'm gonna actually create a net new playground,

95. 19:41

    and I will load in one of these prompts.

96. 19:47

    So again, here's my first prompt. Uh, my first prompt has a, a model associated with it. What becomes really cool here is the ability to, to like, to iterate on the underlying model.

97. 19:56

    Right? I think a lot of us are-- we have access to a lot of underlying providers, and we wanna be able to understand if I change this or if I, you know, a, a provider a-adds a, a new model, what is the impact to my application?

98. 20:09

    So I can duplicate this prompt and maybe change this to GPT four one. I can run this. Oops.

99. 20:19

    Before I run it, I have to add all of my components. So I'll add my, my dataset, and then I can add my different scores that I've, uh, that I've configured here for my change log.

100. 20:29

     And so I can click Run, and then now we'll understand here what is the effect of changing the model, uh, for this particular task with the scores that I've configured against this dataset.

101. 20:39

     So this will churn through all of these in, in parallel, and we'll start to get some results back. Uh, lots of different ways to actually start to look at this data.

102. 20:47

     I always like coming over to the summary layout because I, I can understand, like, um, you can see over here, this is my base task right here, and then my comparison task.

103. 20:54

     So I can understand, uh, looks like, you know, on average, uh, the, the base is performing a little bit better than my, uh, comparison task on my completeness score.

104. 21:05

     Uh, it's faring a little bit worse on my accuracy score. Uh, both of them are, you know, zero percent on my formatting, so probably have some work to do there.

105. 21:12

     But you can start to see how you can use this type of interface to iterate very quickly. Right? Now, uh, the other thing that, um, maybe shouldn't do, but, uh, I can't resist because we just released this today, is this new loop feature.

106. 21:27

     So imagine you are, uh, you know, you're a user within Braintrust, and before this, you would sort of manually iterate here, creating net new prompts, uh, making modifications, changing the model.

107. 21:38

     What if you could now, uh, utilize AI to go and do that for you? So any sort of like cursor-like, uh, interface, we can ask it to optimize a prompt.

108. 21:48

     And I think the really unique thing here is it has access to those, uh, those evaluation results. And so when it goes to go, uh, change that prompt, it understands that-- It changes the prompt, it runs the evaluation, it understands if it got better relative to the scores that we defined.

109. 22:04

     So you can see it, it's gonna go through here. It'll fetch some eval results. Uh, you'll probably see a diff here very, very soon. If we don't, I won't, I won't hang out here too long.

110. 22:12

     But I do wanna highlight one of the things that we are releasing that really enables our users to iterate in a really, uh, really fast way. So here's my, my change.

111. 22:22

     We can click Accept, and then it'll actually go out and run that eval again. Or it would. Uh, I think I, I have an issue with my Anthropic API keys.

112. 22:31

     But the idea here again is, like, we can create that very rapid, uh, iterative feedback loop here within the playground.

113. 22:39

     The other thing here is, uh, we can run these as experiments. So this is, um, this is where we can start to s- create those snapshots in time of that eval, and again, see as I make these changes to that, that application, how has it sort of performed over time?

114. 22:54

     I wanna make sure I don't-- I don't wanna go down. I don't wanna like, uh, decrease the performance of my scores relative to obviously the last time it ran, but, uh, looking out over the last month, six months, whatever it is that we're tracking.

115. 23:09

     Cool. So that was very, very, uh, brief sort of intro to like evals via the UI. Right? Again, like just to summarize, we need a task, we need a dataset, and we need at least one scorer.

116. 23:24

     We can pull those into the playground, and now we can start to iterate. We can save these via experiments, uh, and now we c- we have a way in which we can understand how well this application is performing.

117. 23:35

     Right? This is no longer like qualitative. Right? This isn't like, "Hey, I think this got better. That output looks better." There is actual rigor behind this now. Um, customers oftentimes ask though, like, "I don't really wanna use," or, "I'm not gonna use the, the platform as much.

118. 23:50

     I'd rather use this from my, my code base. Is that possible?" Uh, and it is. Uh, so, uh, we have a Python SDK. We have a TypeScript SDK. There, there are some other ones as well: Go, Java, Kotlin.

119. 24:02

     Uh, for the most part, most of our users are using, uh, Python or TypeScript. Here's a just couple examples of what this might look like [clears throat] from an SDK perspective.

120. 24:11

     Um, actually, if you all aren't, uh, opposed to looking at, uh, some code Here's just a, a really basic example of, uh, defining a prompt within my, my code base and then pushing it into Braintrust.

121. 24:24

     So just leveraging that, that Python SDK. Uh, another example, actually come over here,

122. 24:31

     creating a score. So you give it sort of like the, the things that it's looking for, but now I've sort of defined this score within my code base. Uh, it's version controlled.

123. 24:40

     Um, also the, the prompts that you create within the UI are version controlled as well. Um, but this is just another way to start to interact with, with Braintrust.

124. 24:48

     So again, scores, we could do datasets, uh, and then you can even do, uh, prompts up here as well, I believe. So here's my eval dataset, here's my ch- changelog to prompt.

125. 24:57

     Again, being able to start from the code base and actually push them into the platform is possible, just depends on the organization where they wanna start. The, the other thing here is, like, this is more on the, like, the, the components of the eval side.

126. 25:09

     So that's that top portion. Define those assets in code, run that Braintrust push, and now you have access to that in that Braintrust library. The other one is actually, like, defining the evals in code, right?

127. 25:19

     So what that looks like is, is slightly different. Um, come over here. So we have our eval, but this is just a class that's coming from our Braintrust SDK.

128. 25:29

     Again, it's looking for the exact same things that I just described, right? A dataset that we can use from, uh, Braintrust itself, uh, the task that we want to invoke, and then the scores.

129. 25:38

     So again, defining this here within, uh, within your code base, certainly possible, and then I can run, you know, a command

130. 25:47

     that actually runs that eval within Braintrust. So from here, go into Braintrust, see the eval running. This is now experiment that I can view over time. So again, like, you saw two different types of workflows here, uh, again, catering to maybe two different personas or, again, the way in which, uh, organizations wanna work.

131. 26:06

     It's up to them. Braintrust is very flexible in how we allow our users to, uh, to consume or use the platform.

132. 26:22

     Uh, probably jumped ahead a little bit, but y- this is sort of a recap of what I just showed you. Again, from your code, you, you create your prompts, your scores, your datasets.

133. 26:31

     You can push them in there. Uh, maybe just importantly to-- important to highlight here of, like, why you would do this. Uh, you wanna source control your, your prompts.

134. 26:40

     Uh, the big one here to call out is the online scoring. Uh, uh, I have a section in a little bit diving a little bit deeper into that. Uh, but if you want to use those scores that we define in the dataset, we should push them into Braintrust so that we can create online scores.

135. 26:53

     We can understand how our application is performing in production relative to those scores that we want or that we're using within our offline evals.

136. 27:02

     What I just showed you, maybe another, uh, another variation of that, that eval within our code, again, defining that dataset, defining that task, defining those scores, becomes very, very easy to now connect these two things.

137. 27:13

     It's just, again, up to you to decide where you o- where you wanna do this. The other thing to call out here is that this can be run via CI/CD.

138. 27:20

     We do have some customers that, that wanna run their evals as part of the CI process. So understanding in a more automated way, right, the, the score for A, B, and C, whatever they've configured.

139. 27:30

     Has it gotten better? Has it gotten worse? This becomes maybe a check as part of CI. There is, uh, if you look within our documentation, there's a, a GitHub Action example that shows you how you could set this up.

140. 27:43

     Cool. Let's, let's move to production. Moving to production i- entails setting up logging, right? It entails instrumenting our application with, uh, with Braintrust, um, code. Uh, being able to, like, say, "I wanna wrap this, this, uh, LLM client.

141. 28:02

     I wanna wrap this particular function when it goes to call that tool," becomes very, very easy to do that. But so, so why should you do it? Um, I think I've probably said it, said it numerous times here, but we wanna measure quality on live traffic, right?

142. 28:14

     We actually wanna understand how well our application is performing with those scores. Really great to use during offline evals, becomes, uh, our aid in ensuring that we, that we build really good applications, that we're not creating regressions, but also really important to monitor that live traffic.

143. 28:30

     The other really important thing to call out, I think, is that, that flywheel effect that it creates. So we have these datasets that we use to inform our offline evals.

144. 28:40

     It's very, very easy now to take the, the logs that are generated within production and add those back to datasets. This also, um, speaks to some of that human review component, where we want to now bring those humans in.

145. 28:53

     They can start to review some of the logs that are relevant. Like, maybe there's user feedback equals zero. Maybe there's a comment or whatever it is. But, like, they can filter down to those particular things, and as they find really interesting maybe test cases, very, very easy to add those, uh, back to the dataset that we use

146. 29:09

     in our offline evals. So I think the, the feedback loop or the flywheel effect that this, that this creates is one of the really, uh, fundamental value props of, of the platform.

147. 29:20

     So how do we do this? Uh, there, there's a couple different ways. We're, we're first gonna initialize a logger. This is just gonna authenticate us into Braintrust and point us to a project.

148. 29:29

     Uh, you may have seen when I opened up the platform, I had numerous projects inside of there. You can almost think of a, a project as a, as a container for that feature, right?

149. 29:38

     So you probably have multiple AI features that you're building. I wanna have a container for feature A for those prompts, those scores, those datasets. You could certainly utilize those things across projects, but it becomes a really good sort of way to, uh, containerize the things that are important for that feature.

150. 29:53

     Then you can start really basic, right? You can wrap a, a- an LLM client. Uh, so when you saw those, some of those metrics with, like, tokens and L- and duration and costs, uh, just very basically within the, the s- uh, the script or, excuse me, the, the code here, I just wrapped that OpenAI client, and now

151. 30:11

     I'm just sort of ingesting all of that, those metrics into my logs. That's the easiest way to get started. You obviously probably want to, uh, do a little bit more.

152. 30:20

     Again, maybe you want to, uh, understand when, you know, that LLM invokes a tool. So I wanna trace, uh... I can add a trace decorator on top of a function.

153. 30:30

     Uh, I can even use some of the, like, the, the Braintrust low-level, like, span elements to create custom logs, and I wanna customize the input, and I wanna customize the output and the metadata that we, that we log to that span.

154. 30:42

     So again, you can s- you can start very basic with wrapping a client and then go down to, like, the individual span itself, specifying that input and that output.

155. 30:54

     This leads us to online scoring, right? This is-- I talked a little bit about this, right? This is where, like, when our logs are coming in, we can actually configure within the platform those scores that we want to run, and we can specify sort of a sampling rate, so we don't necessarily run that score across every single

156. 31:09

     log that comes in. Maybe it's ten percent, twenty percent, so on. Um, but it, but it creates that really tight feedback loop that, that I've been talking about. Uh, I'll also-- maybe just important to mention, the early regression alerts.

157. 31:22

     So we can create automations within the Braintrust platform. If my score drops below a certain threshold, let's create an alert, uh, with it, with our automation feature.

158. 31:33

     This is just a-- And I can maybe walk through what this looks like instead of showing you here. Uh, the custom views, this is where, like, there's a lot of really rich information within these logs, and it becomes really important, I think, again, for the human review component to, like, filter these down to the things that they

159. 31:51

     care about or the things that anybody cares about. So we can create custom views within Braintrust with the appropriate filters, uh, and then it's very easy for that human to go into what we call human review mode within Braintrust and sort of parse through those logs, uh, the ones that are the, the ones that are gonna be

160. 32:07

     most meaningful to them. Let me, uh, let me connect some of those, those dots there. So

161. 32:19

     again, showing you some code, may be good, may be bad, but, um, I'm guessing there's some technical people in the room that, that don't mind here.

162. 32:27

     So if I look for, um, the... Uh, you may have si-seen in one of those slides there is a, you know, the Ver-Vercel AI SDK. I wanna wrap this AI SDK model.

163. 32:40

     Uh, again, this allows us f- to just create all of those metrics within Braintrust with just zero lift from us as a developer. This becomes really easy to do.

164. 32:49

     Uh, you can also see where I have specified that span itself, right? I actually wanna define the inputs and the outputs of that. The, the reason you would do that is because you have a specific dataset with a structure that you want to ensure maps to that.

165. 33:03

     So, like, when you are within those logs parsing through them, becomes really easy to add those spans back to that dataset. So ensuring that that data structure is sort of, uh, consistent across offline and online becomes really important, again, to create that feedback loop.

166. 33:17

     So this is, you know, very high level of, like, how we can start to create those spans.

167. 33:21

     Um, now that we do, right, we can now go in, uh, the platform and start to configure our online scoring. So this is here just within this configuration pane.

168. 33:29

     I can click Online scoring. I'll just delete-- I'll create a new rule.

169. 33:36

     Uh, so my new rule, and here's where we can add different scores, right? Obviously, I have a few here that I've been using for the offline evals. I don't necessarily need to select all of them, but I certainly can.

170. 33:50

     And then I want to, uh, apply a sampling rate. So I wanna actually give you an example of what this looks like, so I'm gonna do a hundred percent.

171. 33:57

     The other thing to call out here is that you can apply these to the individual spans themselves and not the entire root span. So where this becomes beneficial is, like, when you are invoking maybe tool calls.

172. 34:08

     You're invoking, like, a, a RAG workflow, and you actually wanna create a score on whether or not the thing that it gave back, uh, is actually relevant to the user's query.

173. 34:19

     So we can actually create a score specifically for that and highlight what that span is here. So again, very, very, uh, flexible in how you apply these scores to the things that are happening online.

174. 34:32

     Now, when I come back here to the application,

175. 34:35

     and we'll just run this again, uh, creating that change log.

176. 34:41

     You'll now start to see here within, uh, the logs,

177. 34:45

     this will start to show up, and then you'll start to see these scores be generated. Right. Again, this is where, like, you can now start to understand over time in production, how are these things doing?

178. 34:55

     How are they faring? Where can we get better? Uh, again, now we can connect again, like, the things that are happening in our offline evals with the things that are happening with online.

179. 35:03

     The other thing to, to call out here is the, uh, the feedback mechanism, right? Uh, w-we certainly have, uh, the ability to do, like, human review, but oftentimes you want your users to pro-provide feedback as well.

180. 35:15

     And so this is just a basic example of a thumbs up, thumbs down, and you can even provide a comment here.

181. 35:23

     This can now be logged to Braintrust. So I should see over here my user feedback. So here's my, my comment, and then I have my user feedback score. But now I can also do something like this.

182. 35:36

     So again, maybe I want to filter my logs down to where user feedback is zero. So click that button. I'm gonna change this to zero.

183. 35:50

     Right. I don't have any rows yet like that, but now I can save this as a view, and people who are now using this as human review can filter this down to where user feedback equals zero, and we can figure out what's going on, right?

184. 36:02

     What are the things that, that fell down here within this application that we need to go fix? The other thing I'll highlight here is our sort of human review component.

185. 36:11

     Uh, actually, um, y-you click that, that button, or you can hit just R, and it opens up this, this different pane of your log. So it's a paired-down version of what you just saw there.

186. 36:22

     It's a little bit easier for a human to go through and actually, uh, look at that, that, that input and that output. But you, as a, as a user of Braintrust, can configure the human review scores that you would like to, uh, to, to use.

187. 36:35

     So I have this add something here, so maybe this is a little bit more free text. I have a better score. Again, these are the things that, that you can add to your platform that map to the, the, the, the review-- excuse me, the scores that you want your, your humans to, to add to those logs.

188. 36:55

     Um, just really quick, I'll, I'll highlight some of these things here. Um, this is what it starts to look like when you instrument your, uh, your application with those different, um, wrappers or those different trace functions.

189. 37:08

     Um, I'm able to understand at a very granular level, excuse me, granular level, the things that it's doing, right? So I essentially have these tool calls where it's going out and it's grabbing the commits from GitHub.

190. 37:18

     It's understanding what the latest release is, and it's fetching the commits from that latest release, and now I can generate that change log. But again, the, the really unique thing here, and maybe a, a different example of this,

191. 37:30

     is I can start to score those individual things that are happening. So this is a different, uh, application with a, uh... Nope.

192. 37:39

     With this example. So if I open this up, I have these, uh, this conversational analytics application. So a user can ask a question, it can, uh, return back some data.

193. 37:49

     But this application goes through these various steps. Like, the first step is to rephrase the question that the user asked. So imagine, like, there's this chat history that we can load in as input, and the LLM needs to rephrase that user question.

194. 38:01

     If the LLM does a really bad job of rephrasing this question, everything as a result of this will fall down. Probably not gonna get a right, a right answer.

195. 38:09

     So what I can do is create a scorer specifically for that span to understand how well the LLM did in rephrasing that question. Can also understand the intent that I was able to derive or the LLM was able to derive from that question.

196. 38:23

     Is that right? But you start to think of, like, these- these more complex type of applications that you build, you need to be able to understand the individual steps that are happening, and Braintrust allows for that very, very easily via these scorers, and then being able to apply them not only again while you're in offline eval kind

197. 38:38

     of mode, but also online, right? We wanna, uh, understand these logs and be able to apply these scores at the individual span level. This becomes pretty powerful as well.

198. 38:51

     Um, I think I actually stole from my, my next section,

199. 38:55

     my human-in-the-loop. Kind of walked through this, uh, a little bit. Um, maybe just another, a call-out. If, if you happen to be at one of our workshops on Tuesday, um, Sarah from Notion, uh, who's a, a Braintrust customer, talked a little bit about how they think about human-in-the-loop.

200. 39:10

     And I think it's, it's important to consider, like, the, the size of her organization, a-and what they're doing. Um,

201. 39:17

     she mentioned that, like, she has a special type of role that they use for human-in-the-loop type of interaction, right? There is-- It's almost like a product manager mixed with an LLM specialist.

202. 39:28

     Uh, they're the people that are going through and doing those human reviews. Smaller organizations, she ma-- she, uh, made a comment that was-- it actually makes a lot of sense for the engineers, some engineers to actually go through and do this as well.

203. 39:39

     Becomes really powerful to pair, like, the automation with the human component of this. Like, this is not gonna go away. I think it, it, it adds value to the process.

204. 39:50

     Um, again, I think I just stole from myself, like, why, why this matters, right? This is really critical for the, the quality and the reliability of your application. Uh, it provides that, that ground truth for, for what you're, for what you're doing.

205. 40:05

     Two types of human-in-the-loop, uh, interactions here. I, I walked you through that human review. Um, give me one second. I'll, and I'll call you. Yeah. Uh, the, the two, the-- Excuse me.

206. 40:15

     The two types, the human review, uh, being able to, like, create that, that, uh, interface within Braintrust that allows that user to kind of parse through the logs in a really easy manner, as well as configuring scorers that allow them to add the, the relevant scores to that particular log.

207. 40:29

     And then the user feedback. This is actually coming from our users in the application. Again, being able to create sort of, uh, views on top of that feedback that then power, uh, maybe the human review and then creates that flywheel effect, uh, that we, that we, that we want.

208. 40:43

     That's all I have today. Uh, appreciate you all coming out here and, and listening to me. But yeah, you had a question. [applause]

209. 40:50

     Thanks.

210. 40:52

     Looking at the, the human-in-the-loop, uh, aspect, uh, user feedback as well as human eval component, how are you handling the, the feedback that you get from your customers or even internally when it comes to taking that log data and identifying or, or extracting the intent and using that to modify the models or the different models or the

211. 41:15

     actual model performance?

212. 41:16

     Yeah. The, the question is around, like, how are we using, uh, human review in, like, some of the logs and informing the, the offline eval portion of this. Largely that?

213. 41:26

     Cool. Uh, yeah, one thing I maybe I didn't highlight here is, so maybe back within Braintrust.

214. 41:33

     I'm gonna go back to my initial project.

215. 41:39

     So imagine now, like, we have, we have all of these logs. We've filtered it down to a particular-- Oh.

216. 41:48

     Are we still showing on the screen?

217. 41:59

     There you go.

218. 42:00

     Awesome. Thank you. Um, yeah. So imagine, like, we, we have this, uh, this, this process now, right? Where we're, we're doing that human review. We filtered it down to the records that, uh, are meaningful for whatever reason.

219. 42:11

     It becomes really easy again to connect what's happening within production. So I-- maybe I select all of these rows, or I select individual rows, but I can add these back to the dataset that we're using within those offline evals.

220. 42:22

     I, I think I've said this, like, a hundred times, uh, over this conference, this flywheel effect. This is, like, I think what's missing oftentimes when we're building these, these AI applications and what Braintrust allows for really seamlessly.

221. 42:35

     Yeah.

222. 42:35

     Uh, [clears throat] I have two questions. Uh, the first one is about production. Uh, is it possible to have multiple models in production and compare how they behave?

223. 42:47

     Yeah. I, I don't see why not. Like, my, my guess is in the underlying application, you're swapping them out.

224. 42:53

     Like, uh, having like A/B tests, you know. I can have like two or three or four and easily compare.

225. 42:59

     Absolutely. Yeah. Um, let's see if I have an example here

226. 43:04

     You're able to, to group some of these scores. Uh, maybe this is sort of an example of, of what you're talking about. So like maybe within production we have different models running.

227. 43:12

     Uh, this, this sort of view here allows us to understand like the, the models that we're using under the hood. Uh, and this is just, you know, you could do this within production as well and sort of do that AB testing.

228. 43:24

     Cool. Uh, my, my second question is about humans in the loop.

229. 43:28

     Mm-hmm.

230. 43:28

     Right? Um, le- let's suppose that I have multiple humans and, um, they behave, uh, slightly different as a scorer, as scorers. Do you have anything or what, what is the vision to do with that?

231. 43:44

     Like, is there a way that I can actually compare how they're scoring or something like that, or not really?

232. 43:51

     So different users can maybe have different sort of criteria for scoring. Maybe, yeah, the, the first thing I would say to that is like there, there should be like maybe a rubric for your users who are interacting with human review, so you're not creating that.

233. 44:04

     Uh, you certainly have the ability to see like who is scoring different things within the platform. Um, I'm not sure if you're able to pull that as like a data set to like assess the differences there, but maybe like before it gets to that place, like have a rubric, have a guideline of, of what scoring looks like

234. 44:20

     for your humans.

235. 44:22

     Okay.

236. 44:22

     Yeah.

237. 44:22

     Thank you.

238. 44:23

     Yeah, of course.

239. 44:27

     Hi. Um, so the scorers I'm used to working with for like LLM-as-a-judge-

240. 44:31

     Mm-hmm

241. 44:32

     ... are like, they're relativistic, right? So they can't tell you is the answer relevancy good or bad for a single run, but it can tell you how it care, compares to previous iteration of like the same test set, for example.

242. 44:50

     Um, do you guys use LLM-as-a-judge scorers for online, or-

243. 44:56

     Yeah

244. 44:56

     ... is... And like, how, uh, are, are they relativistic like that, or do you have some way to be like, "This is a good answer, you know, in and of itself for this sample," or, because it's all, you have new data coming in, right?

245. 45:12

     Yeah. I think a lot of our customers who are, are thinking about this are, are like almost doing evals on their evals, like trying to understand did the LLM-as-a-judge actually do a good job there.

246. 45:21

     So like when that actually runs, there's a rationale behind it, and so you can sort of run an eval of those LLMs as a judge. Uh, I think Sarah from Notion in our workshop described sort of a process like that within Notion, but I think that's, that's sort of like where I would aim you.

247. 45:37

     Okay.

248. 45:38

     Yeah.

249. 45:38

     Cool. Thanks.

250. 45:39

     Cool. Awesome.

251. 45:43

     Okay. Are any of your customers doing evals before they launch? Like, I'm working with a government. [laughs]

252. 45:55

     They don't wanna launch until we show some accuracy levels.

253. 46:01

     Yeah.

254. 46:01

     So we're getting our subject matter experts to enter in all the questions that they have-

255. 46:07

     Mm-hmm

256. 46:07

     ... right? And they have huge data sets of thousands of questions-

257. 46:11

     Sure

258. 46:11

     ... believe me, as a government. Um, and then we're using measures like you're talking about.

259. 46:21

     I- do you have a way to do that? Like, I guess it, I guess it's the same, is it?

260. 46:28

     So what you're describing is what we call offline evals, right? This is development.

261. 46:33

     Right.

262. 46:34

     Um, we can actually do this testing before we get into production. This is what I was talking about, like establish that baseline-

263. 46:42

     Right

264. 46:42

     ... using those scores, using that, that data set that you've already, um, created, but this all happens before we get into production, right? And then you can, like one of the things that I heard from, from somebody earlier is like, one of my challenging things of building this AI application is, uh, establishing trust or creating that trust

265. 47:01

     in this thing.

266. 47:03

     Yes.

267. 47:03

     That's part of what this is, right? It's like, it's showing the, showing, uh, those people the scores of that application. So you start to iterate on this thing. Maybe it starts at 20%, then it goes to 30, then to 40, and so on.

268. 47:15

     That, to me, is the thing that you use to create that trust and, uh, create that like groundswell to push it into production.

269. 47:21

     Okay. Yeah. That's what it... That is what we're trying to do.

270. 47:23

     Yeah.

271. 47:23

     So, but I wondered if, I can see the tool does that. Thank you.

272. 47:27

     Yeah, of course. Yeah. Time for one more.

273. 47:31

     Okay. Thanks. Um, quick question. I love the CI/CD po- components. Um, we're trying to build a lot of, um, we're trying to build like ML as a, as a platform for our team, um, so we can get into evals and stuff like that.

274. 47:43

     So how much of, how much of the monitoring dashboard you have in Braintrust can actually be like take the data taken out and post it in a unified dashboard somewhere else?

275. 47:53

     Yeah. All of this is available via SDK. Right? You can pull down experiments, you can pull down data sets. Uh, so you're able to, you know, pull this down.

276. 48:02

     Like we have, we have a customer that is actually building their own UI on top of like the, the SDK itself. Like, so they built their own sort of like components utilizing the SDK and pulling the, the sort of things that we've logged, the experiments that we have in the application into, into their own UI.

277. 48:19

     So certainly possible.

278. 48:20

     Awesome. Oh, great.

279. 48:20

     Yeah.

280. 48:21

     Thanks.

281. 48:21

     All right, cool. Thanks, everybody. [audience applauds] [upbeat music]