How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity

Talk transcript

1. 0:00

   [on hold music] Hey, everyone.

2. 0:16

   Um, hope you are, uh, having a great conference. Um, so I'm gonna talk about, uh, how to run evals at scale and thinking beyond accuracy or similarity. Uh, so in the last, uh, presentation, we, we learned about like how to ar- uh, architect the AI applications, um, and then why evals are important.

3. 0:39

   In this presentation, I'm gonna talk about like the importance of evals as well as what type of evals we have to choose when we are crafting an application.

4. 0:49

   This is a bit about me. Um, so I work as a lead engineer for applied AI for developer platforms at Adobe. Um, I have also co-authored, um, uh, CI/CD Design Patterns book, um, and also involved in a lot of open source work, uh, across the communities.

5. 1:07

   So let's get started. So how many of you have seen this or are active on the Twitter right now? Like, have you seen these kind of patterns emerging? Um, I think this morning there was a talk where this snapshot was again surfaced.

6. 1:23

   So the one of the most important trends in AI application development is evals, because without evals, we can't, uh, we can't craft any AI application.

7. 1:35

   Then, um, how many of you are, uh, developing an AI application, be it a RAG, chatbot, agents, anything? So if you are working on that, you often have the, uh-- come across these kind of questions.

8. 1:47

   Like, how do I te-test applications when outputs are non-deterministic and re-require subjective judgment? Because we all know in LLM world, uh, you can have the different output for the same set of input.

9. 2:00

   LLMs are non-deterministic. Or how many times you wondering like, "If I am changing a prompt, what is gonna break?" Or, "How am I gonna test that?" And then most importantly, when you are developing an application, in order to, uh, measure the performance or accuracy, you need to find out what tools to use, what metrics to me-- uh,

10. 2:19

    use, or what models are best, because models are getting capable day by day.

11. 2:25

    And the answer is evals. So evals is the fundamental approach where you are, uh, writing sort of test cases to measure your AI applications. And why do we-- why do they matter?

12. 2:38

    Uh, because without measuring something, uh, it can have various impacts. It can, uh, impact your business. Uh, you need to measure whatever s- system output is being produced. How do you align your application with system goals?

13. 2:53

    Or one of the important aspect is how do you keep getting better? Because applications are-- you are developing applications day by day, and you need to make sure it is getting better.

14. 3:02

    And then trust and accountability, this is one of the aspects, um, uh, which is very important because whenever you are developing something for a customer, uh, you need to make sure, uh, they trust your application, whatever output is being generated.

15. 3:18

    Now, when we talk about evals, one of the important aspect to focus on is data. So when we think about evals, when we think about the tests, how do we start?

16. 3:29

    So the very first step is starting with the data. Now, how do you get the data? So there are a couple of approaches to get the data. One is you start small, and you start with the synthetic data.

17. 3:41

    Synthetic data means you can generate the, um, you can generate the ancillary data, you can generate the artificial data, and start validating, uh, your a-applications output against that data.

18. 3:53

    Then it's eval-- Uh, when you think about the data in evals, uh, it's a continuous improvement process. It means every time you generate some output, you need to observe the system, and then you need to keep on refining that data set, whatever data set you are procuring.

19. 4:10

    And another aspect is you need to label your data accordingly. So because data is fundamental to writing evals, so you need to-- When generating the data, you need to define your data set in, in a way where it is labeled into different aspects.

20. 4:24

    It is covering multiple flows or application prospects, so things like that. And then you need to continuously refine that. Another, another, uh, approach which I have learned from my experience is you-- one data set is never sufficient.

21. 4:39

    So when you are thinking about eval, you need to think about multiple data sets based on the flows, based on the applications and, uh, whatever you are trying to achieve.

22. 4:50

    Now, when we think about evaluation, uh, what do we think about? So what do we want to evaluate? The answer is everything, but what does that mean? So you need to defi-- start by defining your goals and objectives, and what do you want to evaluate in your system.

23. 5:05

    Then you need to design in a way where you have module-- modules defined for each of the components. You need to optimize your data handling. Um, and I, I noticed I'm mentioning data again and again, but the point is you need to have different data sets for different flows.

24. 5:22

    Uh, you need to test your flows, outputs, uh, and paths. So if your application involves multiple flows, multiple path, you need to evaluate i-in, in all, all paths.

25. 5:33

    Now, adaptive evals. So one of the, uh, previous presentation talked about, like, there is no universal eval. And that's-- that is again, most important thing because your evals depend upon what you wa-- what type of application you want to evaluate.

26. 5:47

    For example, evaluating a RAG application, a typical RAG application is different from code generation.

27. 5:53

    Uh, if you are dealing with a RAG, typical Q&A type of ac-- uh, application, you can define your eval such as accuracy or similarity or usefulness. Versus when you are generating a code, uh, you want to gen, uh, test the generated code against the actual code base.

28. 6:09

    So that is where you need to defi-- uh, measure your functional correctness of the code generated or how robust that code is generated.

29. 6:18

    Then, uh, when you are trying to evaluate, uh, um, agents. So one of the important aspect for evaluating agents is trajectory evaluation, because agents, uh, can take a different path, and oftentimes you need to define which path they are taking in order to execute a flow.

30. 6:35

    There is also multi-turn simul-- uh, simulation where most of these agents are complex and you, you need to check, like, w-when you are having a conversation, like, how do you evaluate that?

31. 6:47

    Then if you are doing the tool call, then you also need to check the correctness or test suite or, like, how they are-- how the data is being generated.

32. 6:57

    Now, another aspect is how do you scale eval? So one, one strategy is you can cache the intermediate results and regression. You need to focus on orchestration and parallelism, like how you are we-- how you are ge-- uh, running your evals, how you are orchestrating them, how you are parallelizing them.

33. 7:15

    You need to aggregate the results, and then you need-- The important aspect here is you need to run them frequently and then improve upon. So one of the, uh, term which is being u-used in industry is measure, monitor, analyze and repeat.

34. 7:28

    So you need to often measure it, you need to, uh, analyze that and iterate on that. Then you need to strategize what you want to measure. So again, depending upon the use case, there are different type of matrices or different type of methodologies you need to adapt to.

35. 7:44

    And then again, use-- There is no fixed strategy to run your eval, so use wha-what fits best. In some cases, uh, you want your humans in the loop to be taking precedence.

36. 7:57

    In some cases, you have, um, uh, automation tests, automation evals running in. There is a fine balance or trade-off between, uh, human-in-the-loop versus automation, like whether you want the high speed versus high fidelity.

37. 8:12

    So again, depending upon what you want to achieve, um, you want to take-- give a fine balance on that. And rely on process over tools. Reason is because tools, again, you cannot automate everything.

38. 8:25

    So you need to define and establish the process, how do you want to run the evals?

39. 8:31

    So these are some of the key takeaways we just talked about. Um, so one is evals are the most important aspect for AI application. Uh, there is a term being coined now, eval-driven development, uh, which is if, if you think about typical software like test-driven development, this is the eval-driven development.

40. 8:50

    Define evals based on the use cases. Uh, you need to focus on positive as well as negative cases. Then focus on the data. That is-- I cannot emphasize enough on, uh, on that.

41. 9:01

    And then remember to measure, monitor, analyze, iterate in a loop continuously. And always take a balanced approach in fidelity versus speed. Uh, if you have any questions, uh, there's a barcode.

42. 9:13

    You can come later and chat with me. Uh, happy to chat more, and that's all from now. [outro music]