Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Talk transcript

1. 0:00

   [upbeat music] Hi, this is Niklas.

2. 0:17

   I'm CTO and co-founder of Log10, and we want to talk about how you can scale reliability of L- LLM applications using, uh, a new tool that we've built. During this year, I think we all can agree that there's been, like, this kind of craze in the industry, and we've been rolling out a ton of intelligence features based

3. 0:37

   on GPT. And we're now kind of finding ourselves in a now what moment because without knowing what good means in a generative setting, it's really hard and risky to evolve your applications, like changing your prompts, configurations, let alone considering going from one model provider to another to more advanced use cases like cell posting or fine-tuning.

4. 1:02

   We want to introduce a new tool today called llmeval that enables, uh, teams to ship reliable LLM products.

5. 1:13

   It, it is command line tool that you can run locally, and with these four, uh, lines of code, uh, you should be good to go. Um, the initialization creates a folder structure, um, and best practices for storing prompts and, and tests.

6. 1:33

   And then this is based on a super configurable system from Meta called Hydra. So you could basically extend it to your heart's desire, and the metrics that we have wired up are in Python, so they could be any logic.

7. 1:50

   Could be call out to all the LLMs, whatever you want. And after these evaluations have been run, you generate some reports that basically gives you, like, a brief overview of how the entire app and all the tests are looking, but still supports flexible test criteria because, like, these models are very fussy.

8. 2:11

   It's very hard to say with a guarantee that it's gonna be one or the other, but it's fairly safe to say that the majority cases, or say three out of five, should pass.

9. 2:22

   And we're gonna jump into command line and taking a look.

10. 2:28

    We're just gonna create a directory for today.

11. 2:35

    And go into this directory and create ourselves a virtual environment.

12. 2:42

    From here, we're gonna install llmeval and initialize the folder structure. What we should be able to see here is

13. 2:58

    a directory structure where we have our prompts. Let's say a simple case could be this, where we have this message template saying like, "What is A plus B?" Only return the answer without any explanation.

14. 3:11

    So in this case, we know that we have to prompt engineer further in order to get an exact output. 'Cause let's take a look at how the test looks like.

15. 3:21

    In this case, we're taking, like, the actual output from the LLM and comparing it with the expected, and this is like a strict comparison. [clears throat]

16. 3:31

    What we had taken the liberty to do is to strip any spaces that might be... come from, from the left, and that's because some models, in this case, Claude, tends to prepend spaces.

17. 3:44

    And so it's things like that that you have to watch out for. Then we have the metric, which could be any metric that you wanna surface in the report, and then the result, which is then a pass or fail.

18. 3:56

    And in this case, we wanna add four and five, and we expect it to be nine.

19. 4:01

    And I'm just gonna try to run this test here and try to re-revert some of the prompt engineering that we did earlier. So I'm gonna re-remove

20. 4:12

    only return the answer without any explanation. And the way you can start it is llmeval run.

21. 4:24

    But if you want to override anything-- If you just do llmeval run, it runs everything. But if you do like prompts equals math, then it's only gonna run the math example.

22. 4:34

    If you do n tries one, then it's just gonna do one sample. By default, we do five samples, so, so we get, like, a better read on the stability of, of each test, but it might be too much for you.

23. 4:49

    But you can override anything. You can find these default settings here in the llmeval YAML. And but let's try to run this and see what happens. And so this ran across Claude, GPT-4, and, and GPT-3.5 once.

24. 5:07

    So we can go in and generate a report

25. 5:12

    and see, like, actually something failed. What was it that failed? So let's take a look at the output here. And in this case, because we've removed our prompt engineering, GPT-3.5 starts being a bit chatty and says like, "Four point five equals nine."

26. 5:27

    Claude does something similar. So it kind of writes out the... writes out the equation. And now I'm gonna try to revert

27. 5:36

    and see... Let's, let's get this in. And we try to run one more time.

28. 5:45

    Great. Now, when we generate the report, it could say some tests failed, but the most recent test that ran passed. So when you do the report, it's gonna generate a summary.

29. 5:55

    It could generate a report per, per run, but then also say, overall, was there anything that, that failed out of these reports?

30. 6:04

    If you wanna go a bit more advanced, let's say you wanna use tools, y- we c- we have an example here where we are generating some Python code, and again, we had to add a number of different, um, clauses to make sure that it only outputs Python.

31. 6:19

    It tends to be very happy generating, um, s- surrounding explanations.

32. 6:24

    Uh, so in this case, we are gonna see whether or not, um, it returns an actual Python program that could be, that could be parsed. So let's try to run that.

33. 6:37

    If you go in and take a look at this report,

34. 6:41

    you can see that these tests actually end up, end up passing our tool use. And to, to round up,

35. 6:48

    we have model-based evaluation as well, where you can test using other models. And so in this case, say with grading, we can go in and define, like, a full set of criteria.

36. 7:02

    Here, we're about evaluating Mermaid diagrams, giving a score between one and five, and the reason, and that, that is also supported in llmeval. One thing about the previous approach is that it takes quite an amount, amount of work to set up these tests and gather your test cases.

37. 7:21

    And one really compelling answer to evaluation has been model-based evaluation, and it's, uh, it's a setting where you have, well, typically a larger model discriminate or kinda grade or be a judge over the output from another LLM.

38. 7:36

    And that makes it so you can get more nuanced output like pass/fail or a grade from one to five or preferences between different options and its reasoning behind it.

39. 7:47

    There's a number of pitfalls, unfortunately, around this approach around biases towards the output from the model itself. If you're sweeping different models, they tend to prefer their own, own output.

40. 8:00

    They're not very good at giving, uh, point scores, saying anything between zero and one, or larger scores between zer- zero and 100. [clears throat]

41. 8:09

    But there are different ways where you can start increasing the accuracy of the kind of feedback that's been generated.

42. 8:17

    And [clears throat] we've been working on this, where you basically start bridging between model-based and human feedback. So instead of removing the human completely from the feedback, you start taking in all feedback that might have been given prior and start modeling it and say, like, if you have all the feedback from John, then we create an Auto John that

43. 8:38

    will start create-- generating feedback per review, um, for any incoming completions. And so in this case here, we have two pieces of feedback that's been already given by human.

44. 8:48

    See here, it was all just, like, a score of five, or here, just, like, a bit more nuanced. But here, we are kind of pending feedback. And if you click this, we have AI suggested an answer to, to this.

45. 9:06

    And that's all I had today. Um, if you want to get started on, um, llmeval, we have our documentation at our usual documentation site. And you can find me at, uh, Niklas Crafoord at, on, uh, X, or formerly, formerly known as Twitter, or shoot me an email at, uh, [REDACTED:email_address].

46. 9:25

    Thank you. [outro music]