LLM Evals That Work IRL

Talk transcript

1. 0:00

   [upbeat music] All right. Hey, everyone. My name's Aparna, one of the founders of Arize.

2. 0:17

   Uh, we do LLM evals and observability. Um, I wanted to do a session going really deep on this stuff because you guys all hear LLM-as-a-judge, and you're probably like, "Yeah, yeah, yeah," but how does it actually work in the real world?

3. 0:31

   Well, we work with some of the top companies in the space, um, who are all deploying LLM applications, and we've seen a lot go well and not go well in the real world.

4. 0:42

   And so even though you've probably seen this tweet from Greg a bunch, "Evals are all you need," you're probably like, "What does that actually mean when you're putting it in the real world?"

5. 0:53

   And I'm gonna demystify a little bit of that and talk about some real examples today. So first off, there's a distinction between types of evals that we should, we should just clarify.

6. 1:03

   First is there's model evals. If you're on Hugging Face, you're looking at the Open LLM Leaderboard, and you're like, "Okay, Llama, 3B, whatever, 7B is better than this because of some MMLU metric," well, they're actually stacking and ranking different models against each other.

7. 1:19

   And these are really helpful, um, when you see things like the needle-in-a-haystack test, um, to understand which model to actually use. But for most of you in the room who are probably building the applications, you probably care more about task evals.

8. 1:32

   And what I mean by that is, is the LLM application actually working, and how do you define evals that actually help you figure that out? So let's talk about how task evals work in the real world.

9. 1:45

   This is probably a review to most of you. Most of the industry is converging around a couple different options; LLM-as-a-judge, user feedback, heuristic-based approaches. Just as a overview for folks who don't know what LLM-as-a-judge is, it's basically when you're using AI to evaluate AI.

10. 2:02

    I take inputs from-- I take the output of my response, I might take the context that it was given, pass it into an eval prompt template, and then I can actually have LLM-as-a-judge come back with an evaluation of how it works.

11. 2:15

    Let's talk about how this works in a simple application. So this is a really common one we're seeing in the ecosystem. Uh, it's a chat-to-purchase type of application. E-commerce applications use this a lot.

12. 2:26

    So the way it begins is the customer asks some kind of question, "Hey, blah, blah, blah, I'm looking for a new Kindle." And there's first this component, call it like a router, that actually is deciding what the customer intent is.

13. 2:41

    A lot of folks actually use function calling for this, and so there's a function call that happens. It determines what the path to send a user down is, and then there's the actual workflow execution branch of what to happen.

14. 2:54

    This is a really, really simple one. In the real world, the applications get a lot more complex. There's a real-- I see probably a couple of these at, you know, e-every week, where basically the user intent gets decided by an LLM call, and then it has to get the user intent correct.

15. 3:14

    This is what they all care about because otherwise it sends users down the wrong path. It can send it down-- If a user's asking you a question about, "Hey, recommend me some product to go buy," but then it sends me down something related to customer support, well, the issue actually you wanna catch is, did it get the

16. 3:30

    function call of determining that user intent correctly? And so when you look at an application like this where there's a router, and routers are, you know, pr-probably the most common agentic type of workflows we're actually seeing in production today.

17. 3:44

    There's a router call, then there's LLM calls, then there's application calls, and then there's maybe even calls to traditional ML models doing in the middle of where you're actually calling out to search.

18. 3:55

    How many of you guys have an application that looks like this or have seen app-- y-you know, have built one internally? Okay. Awesome. Awesome. Awesome. This is, this is kind of where we're seeing a lot of, um, we're seeing a lot of applications being built.

19. 4:11

    It's not just a simple API call and, and here's a response. It's actually built on top of levels. And so as your applications get more complex, your evals are gonna get more complex.

20. 4:24

    And there's levels to this [laughs] is, is kind of the theme you'll hear today. If you're evaluating something like this, you want evals at different levels of your application. There's an eval, most importantly, at the router level to help you figure out what's the path that it should go down.

21. 4:43

    Did it go down the right execution branch? And then within each execution branch, there's often component-level evals that are being done. I'm gonna actually give you guys a demo, so we can show you a real application, and we can show you where something goes wrong.

22. 4:59

    But just to set some context, you will actually dive really deep today into this router eval in applications. Um, the key thing I think to take away is you'll see questions like this from the demo we're gonna give, but users ask, y-you know, questions in the applications, and typically you wanna figure out, well, did it go down

23. 5:19

    the right function call? So in this case, did it go down something like the user asked about details of a product? Did it go down the product details function call?

24. 5:28

    And then there's another kind of implicit one that's often done, which is, did we extract the right parameters for the function call, and do we give it the right parameters?

25. 5:37

    'Cause if you don't give it the right parameters, then i-it doesn't matter if you pick the right function call, it's still not gonna get it right. So these are kind of the two ones I'm gonna actually walk through and show you guys, um, what it looks like.

26. 5:50

    Um, uh-oh, let me make it big screen. Give me one second. Okay. Uh, I'm gonna actually show you guys Phoenix today. Phoenix is our open source product. You guys are welcome to try it out and download it.

27. 6:03

    Uh, this is actually Phoenix Live for my application that I was talking about. And right now, what you're actually looking at is a trace of the application, very simplified trace of the application.

28. 6:15

    Um, this is what a user asked me, "Could you tell me if there is any current promotions for Samsung whatever phone?" And then this is actually the output that was responded from the application.

29. 6:31

    Within this, um, and you can go and you can look through kind of all the different applications here, all the different kind of questions that users are asking, and you can actually see what, what kind of questions that users are asking here.

30. 6:44

    Each one of these, you'll actually see a full stack trace. What's most important here, and you kind of see the one I clicked on, is one where it actually says it got the function call wrong.

31. 6:53

    So I'm gonna actually go dive into that one, and we can go look at it. You can go look at and see, okay, it says it got the function call wrong.

32. 7:01

    It says the user is actually asking about current promotions for this phone. The generated function call is for a product search, which may not specifically address promotions. A more appropriate function call might be the one that directly queries promotions or discounts.

33. 7:18

    We actually do have, within the application, uh, a function call that's available for promos and discounts. Um, might be easier to see that one in the slides. But it didn't actually call that one.

34. 7:31

    It actually called the one that's specifically about product search instead. And so it called the wrong function call, and this is actually one where the rest of the entire execution branch is gonna be off because it got the first call wrong.

35. 7:46

    This is why it's really important to-- If I had to zoom back out to just what do you care about in an application like this? You care about first your traces because you wanna see what the heck's happening.

36. 7:58

    Where is it going down the flow? You care about evals. You care about evals knowing, well, something like where did it get in the application it wrong? And then you also care about, we'll go into this, explanations of the evaluation, so that when it gets it wrong, you get a view of actually where did it go wrong,

37. 8:16

    what to go fix, and what should I actually go do to iterate and improve the application. Um, traces, you want to evaluate it, and then you want to use it to actually iterate on your application.

38. 8:28

    That's kind of the loop that people do as they're building these. I can take this example, I can go add it to my dataset, for example, um, and then I can say, "All right.

39. 8:38

    Every single time I get something wrong like this, I'm gonna go build up my dataset and then use these to now eventually run experiments, and run experiments where I can track and improve."

40. 8:49

    And this is one where I modified the prompt, and I can run these experiments and then continuously iterate to make sure-- Maybe the function description wasn't right. Maybe the call from the LLM wasn't, wasn't right.

41. 9:01

    And so there's all sorts of things you can actually do to improve, but it really helps when you have evals at different levels of the application to be able to...

42. 9:11

    Oh, is it loading? Evals at different levels of the application so that you know where to go focus and where to actually go improve. Um, so with that, I'm gonna actually jump to just some of the best practices we've seen from the ground.

43. 9:25

    Um, so you saw an example of basically a router-based application, function calling evals. There's different types of levels that we see to it-- to applications. Uh, how many of you guys have a chatbot with multiple back and forths sessions basically in there?

44. 9:40

    Well, typically, you want evals at different levels of that, at a session level, often at a trace level, often at a span level. So

45. 9:50

    getting this stuff to actually work in the real world isn't just single eval and we're good. It's, it's often single eval, help me understand an explanation, let me drill down to where exactly, what component.

46. 10:03

    And so these, these levels really help you do that. And what we see folks do is they actually start to do this in iterative phases. They first start off benchmarking the evals when they're building.

47. 10:15

    Um, and then a-as they're actually building the application and they're building each of the different components, they're developing those eval templates iteratively along the application. And then as they move into production, they can actually go monitor it, run it in, you know, run it as jobs.

48. 10:30

    But you're doing this as an iterative process as you're, as you're kind of building. Um, if there's one thing you take away from my talk today, I hope it's actually this slide. [chuckles]

49. 10:40

    Um, evals with explanations are by far what we see real,

50. 10:47

    you know, real people deploying applications finding the most useful in production. A single incorrect, not incorrect, is just really hard to know what to go fix. But when you have something like an explanation like we were looking at, it makes it easier for teams to go, "Okay, here's what I go fix.

51. 11:03

    Here's what I go dig into." Um, and so run your evals with explanations, um, if you can. There's different types of ways you can generate these evals, actually. There's-- If any of you guys are familiar with like, you know, in ML, there's like regression type of models, classification types of models, et cetera.

52. 11:22

    Well, there's different types of evals too. There's numeric score outputs, there's categorical outputs, multi-outputs, multi-class. Um, can I actually-- Maybe this is a fun question. How many of you guys use numerical outputs as your, as your LLM evals?

53. 11:38

    Okay. Okay. A few brave folks. How many of you guys use categorical evals? Okay. Both. Okay. Nice. Um, I'm gonna actually share-- We did a ton of research around this, and we've been sharing about this.

54. 11:52

    But if you are using numerical outputs today, highly recommend you actually don't only rely on them. Um, here's a little research we shared, and I'll, I'll share some results of this.

55. 12:05

    But- Uh, numeric scores, just for people who, you know, need a refresher on it, is, um, you basically have the output of your LLM as a judge be a single number.

56. 12:16

    And this is a simple example. I have a document, one where we've corrupt- corrupted the document with a lot of spelling errors, and one where we've corrupted the document with very little spelling errors.

57. 12:27

    So one of them, the corruption's, like, eighty percent, the other one, the corruption's, like, eleven percent. And we asked the LLM as a judge, "Hey, can you evaluate and tell us how bad of the spelling errors are actually in this document?"

58. 12:39

    For both of them, it actually gave an eval score of ten on it. And we actually noticed this was really consistent across a- all the foundational models. I think Mistral actually did pretty good compared to some of the rest.

59. 12:53

    But, uh, across all the foundational models, it was actually pretty binary in how it did the scores. It was either a zero or it was, uh, or it was either a one or it was a ten, but it was never, like, this linear range of scores that you'd want it to expect.

60. 13:12

    So as you increase the density of corruption, you actually get an increase in the number of scores. It was pretty binary, which kinda just indicated that if you're using numeric scores, it might not be the right way to evaluate because you're not gonna catch the granularity.

61. 13:27

    You know, an eighty percent doesn't actually mean anything. It's not gonna really mean anything different than a ten percent evaluation. Um, so just a little best practices from the ground that we've been seeing as we've been running evals with, with customers.

62. 13:41

    Um, in the last, like, four minutes here, I'll share a couple more-- This is slightly more model evals-related research, uh, for folks to kinda see the latest on, on that front.

63. 13:53

    Um, for folks who have been following the needle-in-a-haystack test, this was a really popular one on-- trending on Twitter, um, recently. Needle-in-a-haystack test was basically we put a needle in a haystack.

64. 14:07

    We hid a fact in some context window. Um, and the context window size can change, but the key thing we were also trying to figure out is does placement in the context window matter?

65. 14:22

    So this is an example where the fact was placed within the first five percent of the context window. This is an example where the context was placed kinda lower down in the context window, ninety percent.

66. 14:34

    And the reason to do this type of research is, well, if you're using RAG, which I'm sure many of you guys in this room are, well, does it matter if what you put in the context window that's the most important part is actually lower in the document?

67. 14:47

    Does that actually impact the final output of the LLM that's given? And turns out it actually does. So we did a lot of pressure testing against a number of foundational models.

68. 14:57

    Um, we do have the latest, uh, from the Opus model. I just don't have it in this deck right now. But this is actually results from Anthropic Claude 2.1 versus GPT-4.

69. 15:08

    Um, [lip smack] the-- Sorry it's hard to read, but the x-axis on the bottom is basically the context window size, and then the y-axis is basically the depth in the document.

70. 15:20

    And, I mean, GPT-4 was for sure better, um, in, in being able to retrieve the fact. But we consistently noticed actually that if you put the fact, especially as you increase the context window, if you put the fact earlier in the context window, it actually had a really hard time almost remembering or retrieving to pull that document.

71. 15:44

    And so it-- we repeatedly, as we ran this, saw this kind of, you know, red's where it gets it wrong, green's where it gets it right, but it consistently has this, like, red block earlier in the context window.

72. 15:56

    So, um, for folks who are actually using RAG, depending on how much information you're putting in the document, it's important to just balance where you place it in the document as well.

73. 16:08

    Um, uh, another couple research results. We, um, also tested not just retrieval, but also retrieval with generation. What do I mean by generation? Well, after you did the retrieval, you can do things like generation on-- It's kinda like the G in RAG.

74. 16:25

    Uh, you actually generate a response after that. So some of the common types of generations were things like, um, from this financial document, round the numbers or map the dates or concatenate the strings.

75. 16:38

    So these are all common types of generation tasks. And, um, again, we stack ranked two different models against each other. This one's actually super interesting because GPT-4, which, you know, at that point state-of-the-art, uh, did worse than Anthropic 2.1, almost four times as worse.

76. 16:59

    And we were really confused at why. Uh, it was really great at retrieval, but it wasn't great at generation. And we kept going back and trying to understand, like, why this is over so many results and, um, talked to the team, and basically we modified something in the prompt that made it so much better. [chuckles]

77. 17:23

    We asked it to please explain yourself and then answer the question. If any of you guys have noticed, but Anthropic's models are slightly wordier, and actually in this scenario, it was more of a feature versus a bug.

78. 17:36

    Because it was wordier, it kept kind of asking itself to-- It, like, thought through the process, and it thought through the process and then answered the question correctly and did the generation at the end, as opposed to GPT-4.

79. 17:50

    But when we asked GPT-4 to actually explain itself and then answer the question, it was able to get a pretty remarkable jump in performance on generation. So, um,

80. 18:02

    uh, hopefully this was helpful to give you guys a view of just, like, different type of task and model evals. If you wanna hear more about this, um, we're actually hosting, uh, an event, Arize Observe, on July 11th.

81. 18:13

    This is, uh, my code for a free ticket. If any of you guys wanna go, there's all sorts of researchers from OpenAI, Anthropic, Mistral, who are all coming to share model evals, as well as builders who are sharing their own task evals.

82. 18:27

    So, um, check it out. Thanks, everyone. [upbeat music]