Building Metrics That Actually Work — David Karam, Pi Labs

Talk transcript

1. 0:00

   [upbeat music] Uh, we're a bit early, but I guess everyone's here, so we will get started.

2. 0:19

   Um, maybe we'll spend, uh, the first few minutes to get people a little oriented. And actually we, we are, we are quite curious, like, what brings people here.

3. 0:30

   Yeah, why, wh- why are you here? [laughs] [laughs]

4. 0:33

   Uh, what, what about evals, uh, did-- do you... Like, maybe by a show of hands, a few people, like, have you done evals before? Okay. And okay. H- have you struggled with them?

5. 0:48

   Is it-- Was it hard? Maybe by, again, show of hands.

6. 0:50

   Away from the speaker.

7. 0:52

   Um, are you-- Well, so people who have not done evals before, what bring, what brings you here?

8. 1:00

   Somebody? Anyone wants to volunteer?

9. 1:03

   Yeah, feel, feel free to raise your hand.

10. 1:05

    All right.

11. 1:05

    Sorry. Should I move away from the speaker?

12. 1:09

    A little bit maybe.

13. 1:10

    A little bit. All right, I'll stand on this side.

14. 1:11

    He's trying to understand, like, how to approach it better.

15. 1:14

    Oh, I, I see. Oh.

16. 1:16

    Okay, so, so trying to understand how to, how to approach evals. Um, for people who have tried before, what have they struggled with? What has been the... Go ahead.

17. 1:28

    I think the struggle for me is to define the metrics that... Because it's hard to... Like, I do, um, tax research equations. So first, it's hard to define what the correct answer is.

18. 1:39

    It can be many different ways. And two is, um, there are end-based business questions and that you can turn into SQL. And how do you pragmatically

19. 1:51

    improve it versus I design unit tests, and I review it, and I do it again, and I feel like that's still very, um, manual and labor-intensive to me.

20. 2:01

    Right.

21. 2:03

    Yeah.

22. 2:03

    Yeah, so the evals can be labor-intensive. Evals can be hard to, hard to set up. Um, they can be painful to get st- Oh. Get started with. Um, any other thoughts?

23. 2:17

    Go ahead.

24. 2:19

    Well, it's like machine learning is dependent upon training data. Agents are depending upon evals to provide feedback from the world to let them know whether or not they're getting it right.

25. 2:30

    And clearly, it's, it's a sophisticated and challenging problem. So, like, how to do science, you have to have experiments. This is like that. How do you experiment?

26. 2:39

    Right. So we are all-

27. 2:40

    I'm just saying it's super important.

28. 2:41

    Yeah, we are all getting pulled into the, the world of data science and machine learning in some ways, even if we, we don't want to. Go ahead.

29. 2:50

    Yeah, sorry. Uh, before engineering, I was a part of quality assurance team. And you know, all these years people wasn't happy that testing, testing takes, I don't know, thirty percent of feature development time.

30. 3:03

    And now with AI, I think that evaluation took eighty percent of feature development time, and [laughs] people even more not happy about it. And we are trying to find ways, uh, also to, to find the shortcuts, but it seems that it's not always possible because each case is so unique that you just can't to reuse some previous works,

31. 3:26

    and you have to create many things from scratch on different levels. And, and because of this unclear nature, pro- provable nature of, uh, evaluation, it's every time you spend such a way-

32. 3:38

    Speak up. The audience can't hear.

33. 3:40

    -to find problem, the more optimized way to do evaluation.

34. 3:45

    Right. So basically, um... You said speak up or speak low? Okay, speak up. Okay. [laughs] Um,

35. 3:55

    yeah, so, so basically, like, um, it's custom sub- subjective evals, like for your specific use case. You just copy-paste something that exists. Much like tests, we used to do testing.

36. 4:07

    Some people did testing, some didn't. But now for, for... in the world of AI, everyone has to do evals. Everyone is spending a lot of time doing evals. Uh, the best practices don't exist.

37. 4:18

    So I... Go ahead, please.

38. 4:20

    Well, uh, I was just going to add that I'm looking first to understand how synthetic data is playing a role there, and especially for novelty like type of problems where you don't have access to real, like real world data to use as a baseline.

39. 4:33

    So how much of this can impact the eval system?

40. 4:37

    Right. Uh, there are like two aspects of evals. There's the synthetic data, and there are the metrics. Metrics are checking, synthetic data is testing it. Uh, how-- what does synthetic data do?

41. 4:49

    How do you generate good synthetic data to test and evaluate? Um, go ahead.

42. 4:55

    Mine goes along with the first comment. We've been trying local prompt on LLM models and use AI to evaluate, but in the end, it's just not accurate. So we have to do it manually and so hoping to learn then what are some acceptable ways to-

43. 5:11

    Right

44. 5:11

    ...[inaudible].

45. 5:13

    Right. Like, uh, au- automatic evals are, are something everybody desires because you don't want, like human beings to sit and do it. Uh, but the options that are, are somewhat limited.

46. 5:26

    Typically, for most things, you can lean on AI to do human tasks. For, for some reason, evals tend to be hard for LLMs to do right. Um, so I guess that's, that makes a lot of sense.

47. 5:39

    I-- we're not gonna solve all your eval problems here. [laughs]

48. 5:43

    I think this is a, on a very hard research area that we would probably continue working on in the next few months. Um, but David and I were, were at Google for, for over a decade, and, uh, we have been doing evals for, for a long time.

49. 5:59

    Uh, we, we've dealt with stochastic application. We both search. We were building search. Search was also similarly stochastic, and most of, uh, the core effort in Google was to do a good job of evaluating where we are and improving it.

50. 6:14

    Um- And some of the techniques we learned at Google, we are bringing in here. Um, so we, we built a product around it, a bunch of technology around it, but really the biggest takeaway I want people to have is the methodology and the stuff that we have learnt, uh, whatever manifestation it takes, right?

51. 6:32

    So I'm hoping that you get to play with some of our stuff, but also to learn some, some good ideas. Um, David, do you wanna add?

52. 6:42

    Yeah, uh, I mean, one, one thing I would add is at, at Google we used to call this quality and evals was part of it, and there was this constant, uh, idea like benchmarking and, you know, exhausting a benchmark and then moving on to the next benchmark.

53. 6:53

    Whenever we work with clients right now, we, we take a similar approach. So like that Chin meant to- mentioned, so much of evals is just good methodology, setting up benchmarks, trying to figure out the metrics that work, calibrating metrics with humans, calibrating metrics with user data.

54. 7:05

    Uh, what I'm trying to say is that it can get arbitrarily complex. Like evals is not like, oh, there's one way to do it and then you're done. It's, it's just part of how you do development, uh, which, you know, everybody's been struggling with because we've all been trying to learn what it means to develop on top

55. 7:16

    of this new stack. But again, today, in today's session, we're just hoping to bring you some of these ideas where like methodologies that make sense, like how to think about your metrics.

56. 7:24

    I mean, people think of like four metrics. So I would challenge you, Search had three hundred metrics. So like you start to think about like how can you expand the scope of how you're doing this work?

57. 7:31

    Uh, and you know, part of what we're doing is trying to build technologies that make it easy to adopt these, these technologies, how to create benchmarks that are interesting, how to like make it harder, um, and then we can talk about, about this as we go.

58. 7:43

    Uh, th- this workshop will be mostly hands-on. Uh, we'll show a few slides and then we'll dive into the code 'cause I think the best way to, to learn about these things is just to, to do them.

59. 7:51

    Um, I think as we go, we'd love to just pass it around and, you know, if people have, you know, again, uh, people are at different parts of journeys.

60. 7:58

    For some people it's like, "Hey, I don't have metrics and I'll just develop my own metrics." Uh, we worked with clients that have metrics, but they wanna, for example, correlate them to user behavior.

61. 8:06

    They have a lot of thumbs up, thumbs down data. Um, so that goes all the way into a feedback loop. So you see like evals is not just one thing related to testing.

62. 8:13

    It goes all the way into your online system and your feedback loops and all that. So but hopefully a lot of the mental models today will help you kinda gauge that.

63. 8:20

    Um, just two, two ish things. The Slack channel is, uh, workshop-metrics, uh, so you can join there on the Slack and then we'll just be posting things and then we can have discussions and continue the conversations, uh, even after the, the, the workshop.

64. 8:33

    And, uh, the second one is, uh, we have a document that will have all the steps, uh, of the workshop, uh, all the places where you can get the code, where you can get the sheet.

65. 8:43

    Just go withpi.ai/workshop. You will land in a Google Doc, uh, so there's a lot of links. The Google Doc also has a link to the slide deck we're presenting so you can have it.

66. 8:52

    We'll keep it online even after the workshop so you guys can reference it. Um, all right, we'll get started to, to keep you guys on time and maximize sort of coding time.

67. 9:01

    Yeah, I, um, there are four of us. A couple o- other people will just walk around if you guys get stuck on any step, if you're having trouble with anything-

68. 9:07

    Yeah

69. 9:08

    ... just raise your hands and then we'll come and-

70. 9:10

    Just raise your hand

71. 9:10

    ... come over. Um, all right, I'll give you a very like maybe five-minute, uh, quick blurb, show a quick demo, and then get started. Um,

72. 9:21

    so can you guys still hear? Okay. I think we went, we went through this, but basically like most people start with live testing. That's like, you know, try it out, see how it works, change prompts, and honestly for many applications you can go pretty far with just that.

73. 9:37

    Like I think AI, AIs have become much quite good. I think agent systems as you, uh, you were saying, agent systems are, are more complex because of multiple steps, things fail more often.

74. 9:49

    But live testing gets you pretty far. Human evals are expensive. Typically, most companies don't bother setting up human rater evals. Some do. Some have subject mata- matter experts. Uh, code-based evals is where I think majority of the people are spending time.

75. 10:04

    They're writing some sort of a code to test some verifiable things that they can and, and people are moving into natural language, like LLM as a judge type things where it gets...

76. 10:16

    This is not a very good task. We'll get into some of it like for, for t- typical decoder models, the AI models, and so you kinda have to fight against, um, the, the, these models are designed to be creative.

77. 10:27

    They're designed to be, um, that's not what you want from a judge, uh, typically. The s- the scoring system, uh, the scoring system is this idea that you, you, you're not trying hard to build a comprehensive set of metrics from the beginning.

78. 10:43

    You start with some correlated signals. Maybe you start with five, 10 signals that you know for a fact are correlated with goodness. They're very simple signals you can easily derive, and then time you build upon those as you see problems, as you s- as you debug.

79. 11:01

    And, and then you test your app, works well or not. You learn from it. You build your application. You measure it again. So that grip makes it a much more of a feedback loop type process.

80. 11:13

    Uh, that's the thing that we're trying to introduce in terms of methodology.

81. 11:17

    Uh, the thing that David was saying-

82. 11:19

    Uh, maybe-

83. 11:19

    Go ahead.

84. 11:19

    Yeah. One, one thing I will add to that is like there's no right or wrong. Like live testing actually gets you a very long way. Um, sorry, we, we have a lot of feed echo.

85. 11:27

    Um, maybe you wanna turn that off and then we can switch the mic as well right now. Let's do that. Yeah. So ju- just to say like it, it's, there's no right or wrong.

86. 11:35

    I think you should absolutely start with live testing. A lot of people use tracing and they just ma- monitor traces. I think a- as your system scales, you do wanna get a little bit more sophisticated.

87. 11:44

    Uh, so you, you start to layer those things in order of complexity. That's a lot of how we talk about quality generally is this idea of like there's complex things, but they give some amount of investment, uh, of some amount of return.

88. 11:55

    So it's like a little bit of an ROI. Some things are very cheap to do and then you do them, and as your system scales they become impractical and then you just layer in more techniques.

89. 12:02

    So there's not a statement that like any of these things are good or bad. You just need all of them. So think of these things as tools in your toolbox.

90. 12:08

    But eventually what you really want is a scoring system. Now how it manifests for you, you should, you should be the judge of it. But like th- this is one thing to get away with, like just keep laying into your tools and with increasing levels of sophistication you get probably to a system that will look like the

91. 12:22

    one we'll develop today. Um-

92. 12:26

    Yeah, and a-as David was saying, like, um, e-evals are such a, such an investment, so they might, might ask like, "Well, why am I spending so much time on just evals?

93. 12:34

    Why am I not building?" And I think one of the things that, that was core to, to Google, and I think we-- this industry is going to adopt this over time, is that evals are actually the only place you're going to spend most of your time because that's where domain knowledge is going to live.

94. 12:48

    Everything else will just work off of those evals. And so, um, so for example, if you have really good evals, you don't have to write prompts. You can write...

95. 12:58

    Like, you can find problems and use meta prompts or optimizers like DD, DS Pi and others to improve your prompts yourself. You can filter out synthetic data and then use that for fine-tuning if you're really interested in fine-tuning or, or reinforcement learning.

96. 13:12

    But you can also use these techniques online. Like, one of the things that you're gonna test to try today is a very simple but very effective technique that almost all big labs use, um, Google used it extensively, which is the, the idea that you crank up the temperature, you generate a bunch of responses instead of just one

97. 13:29

    response. You can think of this as online reinforcement learning. Generate four or five responses, and then score those responses online and see which one's the best one. And you get a, a pretty decent lift just by doing this, like, without actually doing any changes to your prompts or models and so on.

98. 13:44

    So that-- these are the kinds of things you can do, uh, with-- when, when you have a really good scoring system that you can lean on. So you get to try some of these techniques today.

99. 13:53

    Um, but, but the key point is, don't think of evals as testing in the classic sense. Think of these as the primary place where domain knowledge lives. Um, and then, and then one of, one of the things like that I, I think, um, one of the attendees was pointing out is that t- right now, at this point

100. 14:11

     in the AI industry, we have figured out a handful of standard evals, like you can think of simple helpfulness, harmfulness, hallucinations, and that doesn't really get you far enough.

101. 14:23

     That's good enough to just sort of do a guardrail and make sure you're not doing anything wrong. But we're, we're moving into the world where we wanna build really good...

102. 14:31

     Like, for example, if you're trying to build a trip plan, you, you, you're curious about like, you know, how, how to make that trip plan really be perfect. Like, one of the things I, I really hate about trip plans, if they're too-- like, not interesting enough.

103. 14:43

     I'm looking at a trip plan, I wanna be excited about this place. Typically, when LLMs give me a trip plan, it's a very kinda, a very plan-ish, right? So now you're trying to build these sort of nuances in your applications, and that's the kind of stuff if you wanna evaluate.

104. 14:57

     That's where, that's where the industry is gonna go. So how do you build like these much more nuanced evals? Uh, that, that's one of the places where these traditional evals fail.

105. 15:06

     So as, as I went over, like start simple, iterate, see what's broken, and then improve, right? And so that's the other thing you get to try today is in a copilot-like setting, start with something, test it out with a handful of examples, maybe s- generate some synthetic examples, test it out with good examples, bad examples, see what's

106. 15:27

     working, what's not, and then iterate. We have picked a relatively simple example for today because for workshop purposes, we wanted to keep it simple. But you can absolutely go and try it on fairly complex things, and, and there you can see a lot more of the nuances.

107. 15:41

     Um, so yeah. So there are two parts of today's workshop. The first part is, um, setting up your scoring system, and the second part is once you've set up the scoring system and played with it and iterated on it in a copilot, trying to use it in a colab.

108. 15:58

     You would need a Google account and some, some proficiency in working with colabs py-- and Python code. Um, we'll also introduce a spreadsheet component to this so that you can actually try to play around with this in a spreadsheet, um, so that y-you can easily make changes and test things out.

109. 16:18

     Uh, one last thing I wanna hit on before we jump into the, the, the workshop is what, what is the scoring system? What is this idea of scoring system, right?

110. 16:28

     Like, so just to give you a mental framing of it, um, s- ranking is scoring is evals, effectively. Um, well, that's one way to think about it. When Google does a search, what it's trying to do is it's scoring every document and seeing whether this is good or not, and then giving you the best document that's the

111. 16:47

     best for you. And that's not that different from ch-checking and scoring an LLM-generated content. And the way Google does it is by breaking this problem down into a ton of signals, right?

112. 16:59

     So you can imagine you're looking at SEO, you can look at document popularity, you're looking at title scores, whether the content is good or not, um, maybe feasibility of things, um, spam, you know, clickbaitiness, stuff like that, and then brings all of these signals together into a single score that combines these, these ideas.

113. 17:17

     Individual signals that you have are very easy to understand. You know what that is. You can inspect it very easily. It's not a complex prompt with some random score.

114. 17:25

     It's something that you can easily understand. But it all comes together into a single score. And that's the idea we are sort of bringing in. That's what we call a scoring system.

115. 17:34

     At the bottom level, things are very objective, tend to be deterministic, sometimes just Python code. But as you bring this up to the top, it becomes fairly subjective, and you can like bring these together into very complex ways.

116. 17:47

     Yeah. And I would just, uh, encourage you to think about like wh-why this, why, why this kinda-- why this could work, and, and we know it does work. Um- Some of what you will be struggling with with evals is just, like, you're not measuring the things you need to be measuring, so you have a little bit of

117. 18:01

     a comprehensive initi-issue. And so, like, the question is, like, how many metrics do you need to add? So I give the example, Google Search uses around three hundred signals.

118. 18:08

     Now, maybe you don't wanna be that sophisticated, but, like, Google does care about all those things. They care about, like, you know, the score of the site, they care about popularity, they care about content relevance, spam, porn seeking.

119. 18:17

     Well, they have all these classifiers, all these ways that they understand the content to then bring it together. What this gives you is, like, really visibility over your application and more and more ways to marry your own judgment into it.

120. 18:27

     So if, if instead you just go and say, "Hey, is this a helpful response?" Mostly what you're doing is delegating that, that eval to the LLM itself, right? Or to a rater who you're asking, like, "Hey, see if this is good."

121. 18:38

     But when you break this all down, you get some really nice properties where, like, your variance goes down a lot just because you're measuring way more objective things, so things are not, like, going back and forth all the time.

122. 18:48

     And it's very precise because, like, you're-- you get all these things and you add them together to a more high-fidelity score. And when you are analyzing the data, then you can, like, slice and dice by way finer grain things.

123. 18:59

     And that's kinda like why the system tends to work much better. And the best part of it is, as you iterate, you just add more signals. Like when... Th-this now doesn't leave you as like, "Oh, I either have evals or I don't have evals."

124. 19:09

     But, like, rather you just have a set of metrics that you keep adding over, over time as you discover what actually matters about your app. Oh.

125. 19:26

     Okay, uh, how do I do this?

126. 19:30

     Here, I'll hold it. [laughs] [laughs]

127. 19:33

     Um, okay, so I'll just quickly show you a demo of where you're gonna start today. Uh, give you some sort of basic ideas of what the kinds of things you would be doing, uh, and then, and then I'll just...

128. 19:43

     we, we, we should just get started, right? And so this is, uh, this is a copilot that helps you put together evalu-evals. That's the idea. Where I'm starting is basically just a system prompt.

129. 19:53

     This is... This application is basically a simple meeting summarizer. It takes, uh, a meeting script, talk-talking between multiple people, and then generates some sort of a structured JSON in the end, which is a summary with very specific, um, action items, key, key, uh, key insights, and then a title, right?

130. 20:13

     So it's a relatively simple thing. It's easy for you to inspect and see where things are not going, working and not working well. Um, so typically you would start with something like a system prompt.

131. 20:22

     You can also start with examples if you have a bunch of examples, or you can start with criteria itself. Um, and the, the first step is it would try to use this to build your scoring system.

132. 20:36

     Um...

133. 20:39

     Yeah. And, uh, this is doing exactly what was in that slide, which is trying to say, like, from that coarse-grained subjective thing, what are all the smaller things I can, I can suss out of, out of it?

134. 20:49

     Now, this uses a reasoning model. Like, if you drop this into a ChatGPT or so, uh, we just try to replicate that experience where you get these artifacts on the right-hand side, which are your...

135. 20:57

     is your actual scoring system. But if you think you just wanna do it, like, iteratively through your favorite, like, sort of chat interface, you can also do it. Just say...

136. 21:05

     Oops.

137. 21:05

     Go ahead.

138. 21:06

     Can we do it in reverse where we provide examples of input and output and ask it to score it?

139. 21:10

     Yes, exactly right. So right in front there is an Example button. You can start with example. You can give it actually hundreds of examples if you want, twenty examples.

140. 21:20

     Uh, and then it gets into a much more complex process of, like, figuring out these dimensions based on example. Or you can just copy-paste one or two examples in the prompt itself, and it'll, it'll generate it.

141. 21:29

     Um, the... this is just a starting point, by the way. That's the idea. It, it starts you somewhere, and now you're gonna iterate over it, and that's the exercise you will, you will spend time on.

142. 21:38

     I'll show you few things. Like, this is your scoring system. These are your individual, um, these are individual dimensions, is what we call it, or you can think of these as signals.

143. 21:49

     Uh, they're all questions, um, effectively. For example, this is a-- does the output include any insights from the meeting? That's a natural language question. Um, or you can a-have code, um, just Python code that we have generated.

144. 22:04

     You can edit this code however you want. Uh, y- this, this, uh, this, this is actually as simple as what this, this says. It's... When you, when you look at the actual code, the code is effectively just a bunch of questions, and you're sending these questions to our specialized foundation models that are designed for scoring and evaluation.

145. 22:24

     Um, so you'll get to play with this a bunch in the Colab. You can see what the form of it looks like. There's another thing that you would notice, that there's this idea of critical, major, and minor.

146. 22:33

     These are just weights. These are just ways-- ways for you to control what's important and what's not. The combination of this is done through, uh, a mathematical function that, that you can learn over time.

147. 22:42

     So you actually, eventually you would give it a bunch of examples and it'll learn it. But in this particular exercise, you have a little bit more control. And finally, once you have your scoring system, uh, done, uh, there is a way for you to integrate it into Google Sheets.

148. 22:56

     That's the thing that you're gonna play around with today, which is basically taking this criteria, moving it to a Google Sheet, and testing it against real examples. So here you're gonna work with synthetic examples.

149. 23:07

     You'll develop your scoring system, and then completely blinded to this, we have labeled dataset where users are set thumbs up, thumbs down on the summaries. You're gonna apply this, that scoring system and see how well it aligns with real, real thumbs up, thumbs down.

150. 23:21

     Um, so we don't know. Uh, it depends on you. You, you're, you're building your own scoring system, whether it aligns or not. Um...

151. 23:28

     Yeah. And, and th-this is a really interesting point and w-why evals start to get hard. Like, we, we call this workshop like solving the hardest challenge, which is metrics that actually work.

152. 23:36

     So this idea of, like, correlation ends up being really, really important. Metrics that work are not necessarily good metrics or bad metrics. They're either calibrated metrics or uncalibrated metrics.

153. 23:45

     And this is where, like, uh, at Google, for example, we had a lot of data scientists that we worked with, right? Because they were doing, like, all these correlation analyses and confusion matrices and such.

154. 23:54

     So pa-part of the challenge of, of, of good evals is just getting comfortable with the numerical aspect of these things. And of course, again, having a scoring system that dissects things into much simpler things makes it easier to analyze.

155. 24:06

     But you still have to think about those things, like does this actually correlate with goodness? Like, if it gives a high score, is this an actually a good score?

156. 24:12

     And that's a big part of the methodology. But the good news is, as, as Achin showed in the previous slide, once you have metrics you can trust, uh, like almost all of the rest of your stack gets radically simplified as a result.

157. 24:24

     Okay, so how do you use this copilot? It's created this. Maybe a place to start would be, you know, generate an example. This is a synthetic generation, uh,

158. 24:34

     um, happening behind the scenes. It's taking, uh, your system prompt and other information and trying to generate, uh, some sort of an example. And, and then, and then this example that is generated is scored, and you can see how these individual scores work.

159. 24:49

     Now you can start, uh, kind of testing this a little bit more. You can say, um, "Can you generate a bad example?" Um, and then it will, it will try to generate an example that's broken in some particular way.

160. 25:03

     It-- The, the copilot understands what you've done so far. It has the full context. It's using all of this information to kind of like sort of walk you through this.

161. 25:11

     So in this particular way-- like in this particular case, it created something that is missing a bunch of information. But you can even do things like, you know, specific things like, "Can you create, uh, an example that has broken JSON?"

162. 25:27

     Uh, so this is like basically example generation. That's one, one thing you can do here. The other thing you can do is you can make changes to your scoring system itself through the copilot.

163. 25:39

     So you can either go and ask the copilot to make changes to your, um... Anyway, this is a broken JSON example. You can go to the copilot and either ask it to change the Python code itself.

164. 25:50

     You can say, you know, "Can you update the Python code or-- and, and such of any of these things?" You can also ask it to remove or add dimens-dimensions.

165. 25:57

     Maybe you can say, "Can you generate a dimension that checks for the title to be less than twenty words?" And then the copilot can add these dimensions.

166. 26:14

     Go ahead. So let's say we add a feature to the product, the, the trip planner that also now does bookings for you. Uh-huh. So are you then adding that into the- Exactly ...

167. 26:25

     model as like a higher level? So now it's like more of an agent- Yes ... rather than that. Exactly. And, and what do you do? Do you just add it to your- You just ask the copilot and say, "I've updated.

168. 26:34

     Here's some, like, new examples. Can you add new dimensions to test these new examples?" Or, "We have changed things," and it automatically... I mean- It knows what layer? It knows what layer you're...

169. 26:44

     Exactly. Layer. Yeah. The other thing I would say is the copilot is very helpful if you want this sort of human-in-the-loop type process. Uh, but once you get comfortable with the system, what we see people doing is just use Colab to send large amounts of data and let our system figure things out on its own.

170. 27:04

     Uh, I, I find-- By personally, I find it very good to play with my scoring system here every so often to see if it's like working well, like maybe paste an example from, from a user and see how well it's working, and so on and so forth, so you can kinda go back and forth.

171. 27:18

     Uh, but, but most of our clients actually just fire off these, uh, long, long-running processes. Um. Yeah, I think maybe you wanna- So just- ... spreadsheets. Um, so anyway, so that's, uh...

172. 27:31

     So it just created this new, uh, title length thing for you. So you can like sort of play around with this, um, and it'll-- you can change the questions, you can remove dimensions, so on and so forth, right?

173. 27:41

     The last thing I would show before we get going is, uh, one of the things that you would do is we, we, we have pre-filled a spreadsheet in the, in the, in the workshop directions.

174. 27:52

     You will make a copy of that spreadsheet, and we have a spreadsheet integration where you can run our score inside a spreadsheet itself. You see there are other, other sort of places where you can use it.

175. 28:03

     I'm not gonna get into this, but you can use it for reinforcement learning, for example, using onslaught integration. But there are other places you can actually integrate, uh, with PyScorer, uh, if...

176. 28:13

     Not sure if you're familiar with these. But basically, in the, in the Sheets integration, what you are going to do in this workshop is you can actually just copy this wholesale and put it into a new spreadsheet with the examples that are here.

177. 28:26

     What we are trying to do is just copy the criteria. So you're gonna build a criteria. Here's a copy signal, uh, s-sim, uh, icon. Just click on the copy icon and then go to the spreadsheet that we will-- which, which you have, replace the criteria that exist there with this criteria that you've created.

178. 28:43

     And then under Extensions, these directions are all in the, in the docs. I don't have to remember it. Under Extensions, you can go and call the scorer. The scorer's gonna run across about a hundred and twenty examples, and then you'll see a confusion matrix, which shows you how many times there's alignment on thumbs up, how many times

179. 29:00

     there's alignment on thumbs down, and how many times there's no alignment. And then you can play around in the spreadsheet itself, make changes to the dimensions, and see if you can bring the alignment closer.

180. 29:09

     So this gives you a sense of, um... Yeah, so this is the-- this is what a spreadsheet would look like. This is your criteria, um, which is basically just the English form, uh, the, the spreadsheet form of the English that you saw.

181. 29:22

     Can you zoom in? Oh, sorry. Is that better? Yeah. Okay. So like you can see, this is the label, the question. These are the weights. In the case of Python, there's Python code.

182. 29:34

     So this is what your criteria sheet looks like right now. This is the default one that we put in there for you, but you would replace it with your own.

183. 29:41

     This is the data. This, this basically has actually feedback, which is a thumbs up, thumbs down, which we got from users. Um, some of this is fairly complex, some of this is easy, so it's all kinds of different sort of mix of things.

184. 29:52

     And what you're gonna do is you're gonna select these two rows. This is the input, the output. And then in the extensions, uh, under here, you will have score selected ranges, which is gonna create a score.

185. 30:02

     And then you can look at the confusion matrix and see how it works. So that's sort of how... That's one part of the exercise. But you can easily go and make changes here or even test out by making changes to the data, you know, messing up your like, your JSON and seeing whether how that impacts things, and

186. 30:18

     so on and so forth. So that's, that's the first phase of our copilot, the, as of, of, of our workshop, and we'll talk about the second phase when we get started with that one.

187. 30:26

     Go ahead. Yeah, I'm just-

188. 30:34

     Examples or do teams run this on a subset or just, like, how do you do this at scale?

189. 30:41

     Right. So that, we'll, we'll hit a lot of that in the second part of our workshop. We will directly go into Python Colab code. Uh, we have, um... The, the SDK goes through a bunch of this, um, the details and best practices on how you do it.

190. 30:57

     But you will get to play with this o- uh, in, in this workshop. You, you c- you will... Our, our, our scorers are specifically designed for online workflows. These, like, twenty dimensions that you have, they score them all in sub twenty, like, fifty milliseconds.

191. 31:13

     So you can run on a very large scale. You can run it online, um, fairly easily, and we have batch processes and stuff like that set up for you, so you'll be able to play with that, create sets.

192. 31:24

     Um, typically you want to create eval sets which have a combination of hard, easy, medium, that kind of stuff. We're not gonna get into data generation, synthetic data generation that much, but our...

193. 31:36

     Uh, you'll play with it in the copilot. But the actual data generation stuff there, there's actually, um, documentation that we- you can follow up afterwards on how to create, like, easy to set hard, medium sets for your testing purposes.

194. 31:49

     Uh, the best thing is to s- to sample from your logs some number of things and evaluate or just run it online. Majority of this, these kinds of quality checks at Google were run online, whether it's spam detection, whether it's, like, you know, which...

195. 32:03

     what, what decisions to make. That's the ideal place you wanna be. Most people, it's very difficult to kind of like sort of implement at that point. I think the challenge-- one of the biggest challenges with LLM as a judge is it's so expensive you can't run, run online.

196. 32:15

     So that's one of the things that this solves. Um,

197. 32:19

     all right. So... Oh, go ahead.

198. 32:21

     How this is different than LLM as a judge? Are you using additional API models or smaller APIs?

199. 32:26

     Um, so, so we had a bunch of discussion on this. Maybe I'll go very quick- quickly through these slides. Um, so,

200. 32:36

     so the, the, the-- these models are designed for high precision. Like, for example, if you run the same scorer twice on the same thing, it's not gonna give you different scores, exactly the same score.

201. 32:44

     With small variations, keep the same scores, right? It's highly designed for, like, super high precision. It's just the architecture of these, these models is, is desi- is, is, is basically, um, l- very low variance.

202. 32:58

     Um, they... The, the... Part of the reason is that they're using this bidirectional attention instead of, like, the typical decoder models attention. They have a regression head on top instead of, uh...

203. 33:07

     To- token generations, it's not autoregressively generating tokens, which has a lot of, like, weirdness that happens to it. They did a lot of post-hoc, uh, explanation for scores. They'll come up with a score, then they'll try to justify the score.

204. 33:18

     That doesn't happen. The other thing is these models have been, uh, trained on, um, a lot of data, like, you know, billions and billions of tokens, which are only for scoring the different kinds of contents, a coding content, other type of content.

205. 33:31

     So these are-- these generalize really well across, but that also stabilizes them quite a bit. Um, the, the one thing that I would say, which is really nice, is the interface is very nice.

206. 33:43

     It basically just... You ask a question and give it the, the data, and it will answer the question with a score, and then you can inspect the score and understand why, why it gave you that score.

207. 33:52

     So it's a fairly simple interface. There's no, like, prompt tuning and so on and so forth. Because in this particular case, when you're evaluating, prompt tuning doesn't really like...

208. 34:00

     It's not very natural. Like, you know, how do you explain a rubric to, to a model? So these models understand internally why should this score high, why shouldn't they score high.

209. 34:10

     And then these things come together, uh, using, uh, um, a, a fairly sophisticated model which is like a extension of generalized additive model, which brings all of these different signals together, um, by weighting them based on your thumbs up, thumbs down data.

210. 34:25

     So this is a process called calibration, where you give it a bunch of data and it understands what's important, what's not. What should I-- Some-- If something fails, I should fail everything, for example, spam.

211. 34:35

     But if it succeeds, I shouldn't contribute. Those kinds of decisions, it makes those decisions for you based on the data. So that's sort of... It's, it's a much more advanced way of doing evals, and they're sort of ground up built for eval purposes.

212. 34:50

     It gives them that sort of stability that you desire. And the, and the reason they're very fast is because when you use bidirectional attention, you can build much more denser embeddings.

213. 34:58

     So with fewer parameters, you can get fairly high quality scores. Go ahead.

214. 35:03

     Does it work well with other languages?

215. 35:06

     Uh, right now we have trained it with a handful of languages. Uh, we are, we are pretty soon going to release a model, um, with multi-lang- multilingual capabilities. We don't support multimodal yet, but this is in our roadmap, right?

216. 35:22

     So right now it's English and a few languages, and then we're gonna expand it beyond that.

217. 35:26

     Mm-hmm. Um, let- let's kick it... We should kick off the workshop, and then we'll pass it on, and we can just answer all the questions as well. Um, do you just wanna share the doc?

218. 35:34

     I'll just, uh-

219. 35:36

     Again, reminder, it's like withpi.ai/workshop.

220. 35:40

     I'll just share the doc here as well so that you all can see it.

221. 35:43

     Yeah. I'll also put it in the Slack channel for everybody.

222. 35:45

     Um, but please get started and let us know.

223. 35:55

     Um, sorry. Uh, some people maybe have already started working with the colabs, I've seen. Um, but, uh, if... I'll quickly show others, uh, what,

224. 36:07

     um, what, what the second phase of the exercise is about. Um, but of course, we can continue all of this going forward. Uh, how do I project this?

225. 36:19

     Um, so-

226. 36:19

     Um, so the s- the second part of the exercise... By the way, we, we may not have enough time to wrap it all up here, but the second part of the exercise, feel free to do it on your own.

227. 36:29

     The colab is available for you, so you can, you can, you can use this. Um, this colab, um,

228. 36:36

     this, this is a pre-prepared colab, um, and

229. 36:44

     I don't know if you can see this, but basically this particular Colab will take you through these multiple steps. And a lo- lot of people ask me questions about how to use it in code, so I just wanna quickly go over it.

230. 36:55

     Um, this is some basic installation. This is your scoring spec. This is, this is where all of your intelligence is gonna live. Again, this is a relatively simple example, so it's a relatively simple scoring spec.

231. 37:05

     But the way you get this spec is basically over here, you get into code, you copy it, and you put it into your Colab, right? So this is basically how you get the spec here.

232. 37:16

     It's all in natural language spec, uh, with some Python code in there.

233. 37:21

     This is what you'll change. This is what you'll tweak. This is where everything is going to be. And what you're now doing in this particular Colab, which you can literally click through it, and you don't have to do anything more than that, but or you can play around with it a whole bunch, is...

234. 37:36

     One is we have some datasets, public, public datasets on Hugging Face, which has the thumbs up, thumbs down data. This is the same data if you've seen the sheet.

235. 37:43

     That's the same data over here. You-- it-- you, you-- this Colab is loading that data, running the scorer on it, returning your results, um, and then building a similar confusion matrix that you saw there, which indicates how well, well-aligned your scores are.

236. 37:59

     So just getting you warmed up. Uh, then you can use it to compare models. This is where, like, really interesting stuff starts happening, right? So in this particular case, we are comparing one point five and two point five models.

237. 38:11

     You can see that two point five has a slightly higher score than one point five for this particular task. But because two point five is mostly for reasoning, there's not that much of a delta here.

238. 38:21

     If you go to smaller models, uh, like some of the, the, the, the mini models like the Claude Haiku, you could see a-- you'll see a much bigger delta between the quality based on your own scoring system.

239. 38:31

     So now you can eva-- use your scoring system for evaluating different models. What this is doing is it's taking about, you know, ten examples,

240. 38:40

     calling these five different models, generating responses, and then scoring them using our scoring system, right? So that's a model comparison. The other upset is to try different prompts. Um, people change their system prompts, but they're worried about, um, when they change it, what would happen.

241. 38:57

     This is the right way to kinda make sure that you're not regressing. You have your scoring spec. You try different system prompts, see if your scores are going down on your, on your test set.

242. 39:07

     So again, taking ten examples just to demonstrate how you compare them. Uh, here's-- like we, we created a bad and a good prompt just to kinda accentuate this. So like bad prompts getting much lower score than good prompts on this particular task.

243. 39:21

     Uh, this is the one that I, uh, that, that I'm very excited about because it brings you into the online world and how to actually do this online, where you're taking this one particular transcript and you are testing it out with different number of samples.

244. 39:34

     So if you're-- if you use just one sample, which is typically what you do, generate one response with a temperature of point seven, you get a particular score. But as you up the, the number of samples, what it's doing behind the scenes is creating three or four, four of those responses.

245. 39:52

     Each one is a different response, and then ranking them using our-- the Pi scoring system that you just built and picking the one that's best. And what you would see is as you increase the number of examples, you'll see the score steadily going up, the response quality going up.

246. 40:05

     So this is, uh, m-- literally, like you can click through this in the Colab and run through it, and you don't-- won't need anything else. But it is a bunch of options for you to play around with this right now, later.

247. 40:17

     Just wanna introduce this to you guys before, um, before you all disappear. [upbeat music]