Agentic Excellence: Mastering Evaluation of AI Agents with Azure AI Evaluation SDK

Talk transcript

1. 0:00

   [upbeat music] Well, um, welcome everyone.

2. 0:18

   Um, happy to be here, uh, today. I'm very excited. It's a very, uh, hot topic. Um, so I am Cedric Vidal, uh, Principal AI Advocate at, uh, Microsoft, and today we are going to talk about how to evaluate agents.

3. 0:32

   Uh, so for those of you who were in this very room the session just before, my, uh, colleagues, uh, presented, uh, red teaming, uh, which is how you create, uh, data, uh, that tries to, uh, put your AI in a bad situation and, uh, generate bad, uh, content, and try to verify that it behaves correctly.

4. 0:53

   Today, in this s- uh, session, we're going to, uh, look at more traditional, normal, uh, types of, uh, evaluations when you have a data set that you want to evaluate on your, uh, AI agents.

5. 1:07

   Um, so we're gonna see, uh, look at a bunch of things, uh, on how to make sure your AIs are safe. So, I see that people are still coming in the room.

6. 1:17

   Uh, it's okay. Please come in. Don't be afraid. Um, so

7. 1:24

   we don't want that, right? Uh, [laughs] uh, it's, uh, r- AI agents are all the rage. Uh, to be honest, every single day, like even as I was preparing this very presentation and I was trying the latest models and the latest, uh, uh, SDKs, uh, I'm always, uh, uh, amazed at the progress that those agents are making.

8. 1:47

   Um, and, uh, but of course, the more, um, agency we give to them, the more independent they become, uh, the more the risk of, uh, creating havoc, uh, increases.

9. 1:59

   So let's, uh, see how we can make sure that your AI agents behave correctly, and uh, do not, uh, create tha-that kind, uh, of mess.

10. 2:09

    Um, so how do you, um... Nope, sorry. Um, how do you go about, um, uh, evaluating your AI agent? Do you submit like a couple prompts to validate that the models, um, respond correctly and go, "Yeah, well, that checks.

11. 2:26

    That should go," and put it, uh, in production, or do you go about a more, uh, methodical approach? Uh, if you are doing the former, then I have some news for you.

12. 2:36

    You are, uh, in the right place. Um, you need to change something. Uh, it's not gonna, uh, work. If you're in the, uh, latter, then, um, today I have some, uh, frameworks to show to you on how to-- which might help you improve, uh, your evaluation process.

13. 2:53

    So, when should you start doing evaluations? Um, you may be wondering what's the, uh, evaluation, um, or when does it occur? Um, and I mean, if you have already built an app and, uh, you're asking yourself, "Should I evaluate now?"

14. 3:09

    well, uh, good news, I mean, or bad news, uh, you're a bit late. You should have started way earlier. Evaluation starts, uh, at the very beginning of your, uh, AI development, uh, project.

15. 3:21

    Um, uh, the sooner the better. So to get a b- uh, a sense of how to approach the, uh, subject of a- AI agent evaluation, we distinguish four, uh, layers.

16. 3:37

    Um, uh, we... First you have the model and the safety system, uh, which are platform-specific, uh, level protections, and this is built in Azure. You don't have to do, uh, anything about that when using Azure, uh, uh, models on Azure.

17. 3:52

    Uh, and then you have system message and grounding. So for that part, and user experience, and for that part, that's where your app design matters the most. Um, the key takeaway, the foundation model is just one part.

18. 4:06

    Um, real safety comes from layering, uh, smart mitigations at the application layer, and we're gonna see how to do that.

19. 4:16

    The first thing you should do is manu-manual model evaluation. So which model would, do you wanna use for your AI agent? Um, uh, you wanna get a clear sense of how different models would respond to a given prompt, uh, something automatic metrics can sometimes miss.

20. 4:35

    When you, uh, launch a batch, uh, of metrics, of evaluations on a data set, you, sometimes you have a, a big average score and you might be left wondering, "Okay, but, uh, I'm not sure exactly how it works specifically for a very specific example."

21. 4:53

    Before evaluating at scale, you need first to, um, uh, cherry-pick and look at specific examples. So now I'm going to demo to you how to do that in VS Code.

22. 5:12

    So, um, the first thing here, I'm gonna look at my history,

23. 5:19

    um, is that you can, uh... So in VS Code, there is a new freshly re-re- relatively new plugin, uh, called AI Toolkit, which was released, uh, at Build, I believe.

24. 5:30

    And oh my God, I love that plugin. Uh, before I used to go to, uh, different websites all over the web to, uh, evaluate models and compare. I mean, you had, you had, uh, GitHub Models, uh, but now you can do it right from your development environment, and if you're like me and you like to code, that's

25. 5:46

    where I like to do things. Um- AI Toolkit? AI Toolkit, yes. Um, and so you can ask, though, uh, I did ask it, uh, that question already. "What's a good panna cotta recipe with salted caramel butter?"

26. 6:00

    Which is my favorite. Uh, and then you get, uh, a pretty good response with 4.1. But what if you wanna compare with 4.0, for example? So-

27. 6:10

    What, what's a good recipe for, uh, panna cotta, uh, with salted caramel? Better.

28. 6:24

    And then you can see, uh, side to side how the two models will respond. Uh, 4.1 on the left and 4.0 on the right. And as you can see, 4.1, uh, is a major improvement in terms of, uh, throughput.

29. 6:39

    Uh, you're gonna get the answer much faster. Uh, when it comes to the quality, uh, of the answer, um, uh, so I looked at it ahead of the conference, and to be honest, I prefer the 4.1 answer.

30. 6:49

    4.0 is not too bad, but I mean, 4.1 is so much faster that usually that's what you're gonna use. Um, so that's for

31. 7:00

    spot-checking, uh, the answer of a foundation model, um, without any customization. We, we don't have an AI agent yet. Um, then,

32. 7:12

    uh, you want to evaluate the whole system. So that's where, uh, we are gonna actually build an AI agent and evaluate the, the agent, uh, from a systemat- systemic approach as a whole.

33. 7:24

    Um, once you have selected the model, it's time to evaluate it end to end. Uh, and um, so let's jump in and sh-- let me show you how that works in VS Code.

34. 7:36

    So same. That same AI Toolkit, uh, extension for VS Code. Uh, wow. I mean, to be honest, I love it, uh, because now you can build an AI agent, like, super fast and evaluate it super fast too.

35. 7:48

    So here I created ahead, um, uh, I prepared an agent to extract agenda, uh, and event information from web pages. Um, uh, for me as an advocate, I do that kind of talks pretty often, and I need to know, uh, uh, I created, basically, an agent that helps me easily fetch information from the web and pull, um,

36. 8:11

    the, the names or list of, uh, of talks, of speakers and number of attendees, that kind of thing. And, um, it's super easy to do. So I'm gonna show you how to create, um, a new agent really quick.

37. 8:24

    Uh, and you have an example here with a web scrapper, um, and it automatically generates a system prompt saying, "Hey, you are a web exploration assistant that can navigate websites."

38. 8:35

    Uh, it's gonna configure an MCP server, um, uh, ready to use. And, um, if I run it,

39. 8:44

    uh, it's gonna start, uh, the Playwright MCP server. Um, uh, the-- by default, the, uh, it uses an example domain.

40. 8:59

    And will extract... You can see the background, will extract information about, um, the website. Now I'm gonna switch back to the agent that I created, 'cause the one I just showed you is the, the built-in.

41. 9:12

    So this one, um, I created, and I'm gonna use a GPT four one. Um, and this one is more focused. What I want is to extract, uh, the name, date, location and atten- and number of attendees, uh, in a specific format.

42. 9:28

    Um, and, uh, for that website, which is a Luma event page. So run. So what I did is that I took the, the automatically generated, one of the sample, uh, AI agent that was created by, uh, AI, um, uh, Toolkit, and I customized it for my use case.

43. 9:49

    And here you can see the AI, uh, agent, uh, working with, uh, and piloting, um, uh, Playwright, going to the web page, extracting the information and giving me the response.

44. 10:03

    So the, um, the event is AI Agents and Startup Talks at GitHub. Location is, uh, GitHub headquarter in San Francisco, uh, on June eleventh. Uh, and for now, we have two hundred and sixty-nine, uh, people that registered.

45. 10:16

    And I hope that after doing the demo, we're gonna have more [laughs]

46. 10:20

    uh, because that's an event that I co-organize, um, uh, in San Francisco. Um, and, um, so now that we have, uh, spot-checked our, uh... We have built, we have customized, we have spot-checked what our agent, AI agent does for a specific, uh, input.

47. 10:39

    Let's see how we can evaluate it on multiple, uh, inputs. So you have, uh, uh, a, um, a tab here called, uh, Evaluation, uh, which allows you to, uh, take, uh, that AI agent previously, uh, configured and to execute it for, uh, on a data set.

48. 10:58

    So here I, uh... So I can type Run all.

49. 11:07

    And in the background, it's gonna run, uh, the agent of those inputs and give us the answer in the response column. As you can see, I had executed it, uh, before, so you can see, uh, what was the previous answer.

50. 11:20

    Uh, but what's cool here is that you can take that answer, have a look at it, and as you can see, we can see the, uh, the information correctly, um, extracted.

51. 11:29

    Uh, what's interesting is that the web page here, by the way, does not contain the number of attendees. Still, we can see here that we have a, an answer here.

52. 11:38

    That's very interesting because it actually went-- well, it went to the, um, Reactor, uh, page, so that event page, found the link to the Luma page, navigated to the Luma page and on the Luma page we have the number of attendees.

53. 11:52

    So it pulled in, it mixed the information from the Reactor event page and the Luma page, uh, to co-collect everything I needed to, in order to get my answer.

54. 12:03

    Okay, so that was a side note. Um, and I mean, I love it. In both cases, those are good answers. So we can, um, manually evaluate whether it's a thumbs up or a thumbs down, and then we can do a few things.

55. 12:18

    We can, uh, export- The data set to a JSON file. Uh, so I'm not gonna do it, but, uh, it's basically a JSON line file with, uh, the result of the evaluation that you can then re-inject, um, into, um, a, a, a more automated, uh, system.

56. 12:35

    And then once you have your mo- your agent like this, you can type view code, generate, uh, using whichever framework you prefer. Uh, OpenAI Agents is usually the one people wanna use those days.

57. 12:47

    Um, and then you have all configured, uh, an agent with, uh, the MCP, uh, server and, uh, boilerplate code to evaluate, uh, uh, to run, sorry, your agent. So let me close that.

58. 13:01

    Let me move on. Okay. So we've seen how to build and, um, manually evaluate, uh, our AI agent, uh, on the spot example, and, uh, how to run it on a batch of example locally, so a small batch.

59. 13:16

    Then how do you scale beyond a few samples?

60. 13:21

    Uh, let me move on to the next slide. Where do we get the PowerPoint presentation? Sorry, what? Do you get the, do we get the PowerPoint presentation? Yeah, sure. [laughs]

61. 13:32

    I can share it before. Excuse me. I'm... [laughs] I have five minutes left. Uh, we, I, yes, I will share it. Um, so, um,

62. 13:40

    okay, so we've seen AI Toolkit. Okay. So how do we scale, uh, beyond what I just showed? Uh, because, okay, uh, eyeballing, uh, is great to get a sense if it works, but what you wanna do is, uh, uh, go through more thorough, more wide, uh, range of checks, uh, and you want to automate this.

63. 13:59

    Um, so, um, well, Azure AI Foundry gives, uh, a wide set of built-in evaluators to automate and scale those evaluations. We've got AI-assisted quality checks like groundedness, fluency, coherence, perfect for measuring how well your agent performs in realistic conversations.

64. 14:17

    You also find classic NLP metrics, uh, F1, BLEU, Rouge for benchmark comparisons, as well as, um, uh, a suite of AI-assisted risk and safety evaluators, and you can also customize and build your own.

65. 14:35

    Um, once you've spot checked, uh, the next che- the next step is to scale. Uh, and for that you, uh, you need automated evaluation, um, to measure quality at a bigger scale.

66. 14:48

    Uh, you can do it either in Azure AI Foundry portal or via code, and I'm going to show you how to do it via code. Um, it's important because, um, we can define what we want to measure based on our app's, uh, use and goal.

67. 15:01

    Um, so now demo. Crazy how twenty minutes goes fast. Um, so here is a notebook, and given the time we have, I'm not gonna execute it because it takes a bit of time.

68. 15:14

    Uh, but here you have the, the Python code to, um, and I'm gonna share at the end of the presentation the link to the notebook. Uh, you have the notebook that allows you to, um, programmatically, uh, use, connect to the, an Azure AI Foundry, uh, project, uh, and run those evaluation.

69. 15:33

    So the key function here is that you can define... So those are quality, uh, evaluators to evaluate relevance, coherence, groundedness, fluency, and similarity. And you have an evaluate, uh, function code that takes those evaluators, takes the data set that you want to evaluate, uh, and, um, bulk evaluate, um, the AI agent on all those, um, uh,

70. 15:56

    metrics. And the result is, uh, here. So, uh, on that data set here, uh, which is about camping, uh, like what is the capital of France, which tent is the most weatherproof, what camping table, what- whatever.

71. 16:12

    Um, you can see for each, um, question, uh, here, you can see, um, the result of the evaluation, which also you can configure a threshold. So it's gonna get, um, give you an answer between one and five, and depending on which threshold you configure, because depending on your application, you might, uh, want your AI agent to be

72. 16:32

    more or less strict, depending whether you're in the gaming industry, where usually, uh, they accept more, uh, like, um, violent content, or whether you are doing an application for kids.

73. 16:43

    Obviously, the threshold is not gonna be the same. Um, and so I'm gonna move on to the next. Um, so the, in this case, this, this was passing. I wa- I wanna just wanna show the next, uh, evaluator that we have.

74. 17:00

    Um, uh, also very cool. Now you can evaluate multimodal models mixing text and images, uh, this is very, um, important, and for multi-turn conversations. So here, um, I have an image on purpose.

75. 17:20

    Uh, so I tried to find a violent image, and it's hard to find something violent that you can show at a conference publicly, right? So, uh, I, I did what I could, and I spent a lot of time looking, and believe me, when you search for something violent on the web, you see things you don't wanna see.

76. 17:35

    Uh, and so I found that, um, and let's go straight to the end and see what it tells us. So, um, the system's response, uh, blah, blah, blah. The image actually depicts a character with numerous pins or nails protruding from the head, which is a graphic and violent depictions.

77. 17:54

    Uh, but what's interesting, the score is four. It's not like five. It's not the max. So it's failing, um, but it, um, like for example, if you were doing like a, like I said, a, a video game with violent content, you could increase to four and say, "Hey, four, I'm fine with it."

78. 18:10

    Uh, and so in order to be able to generate that kind of image.

79. 18:15

    Uh, and at the end, uh, what's interesting,

80. 18:19

    uh, and I'm gonna show you on another... Uh, okay, I'm gonna move on. Um,

81. 18:28

    I showed you that. Um, okay, I don't have...

82. 18:35

    I wanted to show you something else. Okay, you also have an evaluat- evaluator to, uh... Oh, I think I'm on time, sadly. Okay, so here's, um, some links to more information.

83. 18:51

    Uh, we have, um, uh, on GitHub, Azure AI Foundry discussions, uh, where you can come and ask questions about, uh, that evaluation SDK and how to build a AI agent and how to evaluate them.

84. 19:04

    Uh, you have the Azure AI Foundry Discord too, where you can come and discuss if you prefer Discord. Uh, and then at the very end, you have my, uh, contact, uh, information, uh, if you want to reach out for more, uh, questions.

85. 19:17

    Um, so yeah, very packed. Uh, sorry, a lot to say and very little time. So thank you very much. Uh, I'm here if you have more questions. [clapping]

86. 19:29

    How are you sharing the slides? Uh, that's a good question. Uh, I'm gonna put them on, uh, the Discord.

87. 19:38

    Where? The Discord. Where? On the, on the middle, you have our Discord server, so you can come on the Discord server, and I will post it there.

88. 19:53

    Thank you very much. [outro music]