Building Code-First AI Agents with Azure AI Agent Service — Cédric Vidal, Microsoft

Talk transcript

1. 0:00

   [upbeat music] Well, um, this is very exciting because, uh, I mean, twenty twenty-five is clearly the year of agents.

2. 0:20

   Uh, compared to, uh, the past two years, things have moved so fast. We went from very simple, uh, prompts, which were already incredible, but now, uh, we move to, uh, the next step, where we have, uh, agents that can autonomously, uh, achieve, uh, goals, um, without us knowing exactly how they do that, which is quite incredible.

3. 0:43

   Um, but the question is: How do you make them? Um, so that's what we're going to see today. Um, I am Cédric Vidal. I am a principal AI advocate at Microsoft.

4. 0:54

   Uh, you can find my information down below. Um, and today, I'm gonna be, uh, your host, and, um, I am gonna be helped, uh, today by our proctors, uh, by Mark, uh, and, uh, I'm sorry, I forget your name.

5. 1:09

   I feel terrible. Nagmar. Nagmar. Did I say it correctly? Yep. Okay, thank you. Um, so, um, a big thank you to you two, uh, to, uh, help, uh, help me today.

6. 1:21

   So, um, during the workshop, uh, if you have any question, please raise your hand. And, um, Mark or Ar-- or Ark-- [clears throat]

7. 1:31

   Argmar? Nagmar. Nagmar, sorry, will help, um, uh, will come and help, uh, answer, uh, questions. I feel super bad. Um, anyway, so [chuckles], um, today, like I said, um, I'm going to set the scene first.

8. 1:49

   So what are we going to, uh, to do today? So in order to, um, put our hands on, uh, the keyboard and, uh, create an, uh, and show to you how to create an agent, uh, we're gonna use that use case.

9. 2:03

   Um, imagine that you are, uh, working for an outdoor, um, and hiking equipment company that sells, uh, equipment online. Um, we are going to... So what you wanna do is y-you need to, um, uh, build a system that allows to analyze your sales, uh, data mixed, uh, with product information, uh, generate ad hoc diagrams.

10. 2:27

    Basically, a UX that, uh, c-- your salespeople can use, uh, very easily, uh, where we move away from the old, uh, paradigm, where we had to hard code every single use case, every single view, every single query.

11. 2:43

    Where now, those, uh, the database, the, the queries are going to be generated automatically. The UX is going to be generated automatically to accommodate the type of information that you are displaying.

12. 2:56

    And we are going to see how to create such an application.

13. 3:02

    Um, an agent. Okay. Um, w-what is it? Um, because, like, the definition of an, of an agent has changed so often. L-let's be honest, uh, even the specialists in the industry don't agree exactly on what they are.

14. 3:18

    And even the definition of what an agent is has evolved over time over the past three years, uh, as people have got more acquainted and we're discovering what we could do with it.

15. 3:28

    You could-- You would imagine that a definition should be set in stone, but in that case, it's been difficult to agree. But what we're gonna... The definition we're gonna use today, um, is that, um, it's, um, semi-autonomous software that, uh, to which you give a goal and will work to achieve that goal relentlessly, um, using

16. 3:53

    tools and information that it can pull from databases, uh, and data, data stores, uh, at large, and iterate until it achieve that goal, so until the system stabilizes and the goal are met.

17. 4:08

    That's the general, uh, definition. But we're gonna see that, um, depending on the context, they can be, uh, more or less simple or complex. Um, in order to do that, an agent should be able to do three things.

18. 4:24

    Uh, re-- do reasoning, um, over a provided context to provide a cognitive function such as, uh, deduction, correlation, understanding cause and effect, uh, all that, uh, domain of cognition, um, uh, the LLM now has proven that it was able to do a lot of those.

19. 4:41

    Not perfectly, but it's getting better every day. Um, the second one is integrate with data sources for context, and the last one is act on the world. Because in order to stabilize the system, in order to be, uh, useful, like before the first generations of, um, uh, LLM-powered system were just about pulling information and displaying it, but

20. 5:02

    now we are moving a step forward where we are acting on the world and modifying the environment until it stabilizes and reaches, uh, the, uh, expected goal.

21. 5:13

    Um, ta-dum. Um, so what kind of application are we going to build today? It will look like this. Um, so this is a screenshot of a slightly different application.

22. 5:25

    What you're gonna build today does not look exactly like this. But the idea, uh, is that you can ask a question in plain English, such as, "Show the sales of backpacking tents by region and include a brief description in the table about each tent."

23. 5:39

    And it's gonna, it's going to pull information from the database, uh, as well as the product information, and mix all that information together, reason about it, and display the content.

24. 5:51

    And the shape and form of the, the display of the UX will be, will depend on the type of information which is requested. Here is a table-

25. 6:01

    Here is a pie chart. Uh, because the question here, yeah, create a pie chart of sales by region. So, um, the, uh, the system is going to understand that we want to create a visualization.

26. 6:16

    Okay. Um, what technology are we, uh, going to use today to build our system? We are going to use Azure AI Agent Service. Um, the reason-- Um, so before I, I, I dig more into the details of what this is, um, you have so many ways to build an agent today.

27. 6:35

    So many frameworks: uh, LangChain, LongGraph, uh Semantic Kernel, uh, and so many others. Um, the Azure AI Agent Service, um, has, um, the advantage that it's, uh, stateful and quite easy to put together because usually when you build any kind of LLM, uh, application, I don't know if you're aware, but it's stateless.

28. 7:00

    You need to manage the state client side, so it's the responsibility of the application developer to store the conversations, to, to, uh, and to handle all the logic of pulling information from various systems, as well as executing the functions, the tools.

29. 7:16

    Agent Service moves all the responsibility to the cloud, uh, on the Azure, uh, platform, and everything is managed. So, um, it comes with pros and cons. We're gonna see what they are.

30. 7:26

    Uh, but w-when it-- about, uh, the pros, um, the big advantage is that it provides, um, a very, um, simple development, uh, workflow because all the state and the context and the agent configuration is managed in the cloud.

31. 7:42

    Uh, the integration with back-end data, uh, and data sources is also managed by the, in the cloud. Even if, if you may also mix and match. You can mix, uh, things that are managed, um, by Agent Service in the cloud with things that are managed locally.

32. 7:57

    It's possible. Um, and it supports all the model families that are supported on the, on Azure AI Foundry model catalog. Uh, plus the Microsoft, uh, Enterprise security, uh, which is very well known to be very, uh, robust.

33. 8:13

    It's sometimes a bit difficult to set up, but, uh, that's the price to pay for security.

34. 8:21

    So the application that I just showed was using, um, Chainlit. The one we are going to build today is gonna be Command Line. It's gonna be, uh, slightly easier.

35. 8:31

    Uh, but so basically at the top you have, um, the, uh, application layer, uh, with the framework. So here, Chainlit. In our case it's gonna be Com- uh, Command Line.

36. 8:42

    Very simple, very basic, uh, with a query function, which is going to use Azure AI Agent, uh, Service, uh, with instructions and, and models

37. 8:55

    and, uh, actions. So, uh, for function calling, and I'm going to explain what it is. A code interpreter, a file-- And I'm also going to explain what it is.

38. 9:04

    File search and grounding, which being searched for, uh, web, uh, information ground-grounding.

39. 9:11

    And we're gonna go through each one of those, uh, during the workshop.

40. 9:17

    So like I said, um, Azure, Azure AI Agent Service comes with pros and cons. Uh, the pros is that it manage everything for you. That's a big pro, right?

41. 9:30

    Uh, the con is that you need to understand the diagram. [laughs]

42. 9:33

    Uh, more or less. Uh, you don't need to understand all the details, but, um, one of the... So I'm going to change slide. Um, so the first thing, what is...

43. 9:45

    You, you're gonna have to follow a sequence of steps. Um, and you have quite a few steps to follow in order for agent to work. Um, first you need to create an agent.

44. 9:57

    Once you have created your and configured your agent, your agent exists in the cloud. It's kind of weird at first, uh, because when you're used to stateless, uh, way of doing things, um, that stateful programming model is not so common those days anymore.

45. 10:14

    Uh, but it becomes, um, relevant again in the age of agents. Um, so once you have your a-- Which means that if you're, you want, you have an application that wants to reuse an agent, if you have created the agent before, you need to reconnect to an existing agent.

46. 10:29

    It's kind of like in SQL, like the create or, uh, or update a table. You only create the schema if it, if it does not exist, uh, yet. Um, so it's kind of a create or update agent for most applications.

47. 10:42

    So you create the agent, then you create a thread, or you reuse a thread. Uh, then you run the agent on the thread. Then you check the run status, and then you display the agent's response.

48. 10:55

    Those are the big, uh, steps that you need to get familiar with when building with Azure, Azure AI Agent Service. Um, then you are going to configure instructions. Um, those instructions are going to be attached to the agent, and they are stateful.

49. 11:12

    So they are uploaded in the cloud, and once they are, they are there, you can reuse your agent, and you don't have to send the instructions every single time.

50. 11:21

    Uh, which in terms of bandwidth and network is interesting.

51. 11:26

    Um, then you configure your model, you add, uh, data sources. Um, so one of the pros of using Azure AI Agent Service is that you can attach data sources directly to the agent.

52. 11:37

    And you can do that, uh, either dir-- uh, graphically through Azure AI Foundry, or you can do that programmatically through the SDK.

53. 11:47

    Uh, then you can attach tools. Uh, and we're gonna see how to do that, how we attach each one of those tools. Some of those tools can be client side, so you-- are, are managed by the application code.

54. 11:58

    Some of them can be, uh, managed by Azure.

55. 12:03

    Today we're gonna see, uh, file search, code interpreter, function calling, and Bing search. Um, here is an example of a thread, of what a thread looks like. Um, so the user's message is gonna be, "Tell me the total sales by region."

56. 12:21

    Um, so what's gonna happen is that in order to get the total sales by region, we need first to get the sales. Uh, I mean, we need to query...

57. 12:32

    Sorry, we need to query the sales data store. And it happens that in this case, the sales data store is a SQL relational database. And as you know it, the way to interact is using SQL queries.

58. 12:47

    So we are going to generate a SQL query dynamically, um, depending on the user's request. Um, and then we are going to send that SQL query, execute it on the d- database, get the list of records back, re-inject those records into the LLM, which is going to generate a message in, uh, plain

59. 13:12

    text from those, uh, list of records. Um- Excuse me. Yes. The SQLite is just for an example. It can be any- Yeah, of course. Yeah, yeah, it's an example.

60. 13:22

    In the, in the agent we're going to build today just because it's convenient, but you can of course, uh, connect it to any database you want, relational, the document, uh, you know, an API, like, really it doesn't, uh, matter.

61. 13:35

    Uh, for the ones which are managed locally, for the ones which are managed by Agent Service on the back end, uh, the list is more, uh, restrictive. Uh, and to be honest, I don't have it on the top of my mind.

62. 13:46

    Um- We're not going to see any RBAC or anything. Sorry? We're not going to see any role-based access control or anything. Not today. Not today. Um, it's more, it's a more advanced topic, obviously.

63. 13:58

    Uh, and it depends a lot on your use case. Um, so then show as a pie charts, which is the second question we asked in, uh, in the previous, uh, screen I showed you.

64. 14:09

    Um, this one is, is gonna be quite interesting because we're gonna use a, a tool called Code Interpreter. Um, what it does is that it's gonna take the query, generate Python code, the...

65. 14:24

    And, um, the Python code is going to be executed in a sandbox, in a secure, safe sandbox. Um, and it can be whatever is supported, whatever, uh, s- um, package, Python packages are avail- available in the environment.

66. 14:40

    Usually you use it to generate, um, diagrams. Um, and it's gonna generate the Python code, which is going to generate that visual representation. Uh, it's gonna execute the code.

67. 14:52

    The code is gonna save the image somewhere on the file system inside the sandbox. Then the agent is gonna pull that image out of the sandbox and send it back to the, uh, client application.

68. 15:05

    Okay, so like I... what I said, uh, this is quite a lot, uh, to, uh, digest, right? Uh, but the thing is, you just have to go through and get a mental model of how that works.

69. 15:16

    And once you understand that, uh, you don't have to manage it, uh, yourself, which is quite, um, interesting. Does it use any vector databases behind the scene? Today? Or, uh, uh, do we have a vector database today?

70. 15:31

    Uh, I'm blanking. I think we do. I think we do, but, uh, I'm gonna double-check. Uh, because we ingest the documents, and I think we ingest the documents inside an AI search instance.

71. 15:44

    Um, one more very important thing, function calling.

72. 15:49

    Um, so to be honest, function calling is not new when it comes to LLMs. I was doing function calling, like, literally when the first version of ChatGPT was announced by, uh, asking the LLM to give me, uh, answers from, separated by commas.

73. 16:02

    Uh, and say, uh, generate a function and the argument separated by commas. You, you can still do it by the way. Uh, but nowadays it's much more ef- uh, efficient to have structured output.

74. 16:12

    It generates well-formed JSON. Even the LLMs are optimized under the hood, uh, inside, uh, the data center to generate, um, op- um, JSON optim- uh, optimally. Um, but so what...

75. 16:28

    The principle of function calling, actually the name is bad. I've hated that name ever since it was, uh, coined because it's not function calling, it's function routing. The LLM does not call anything.

76. 16:41

    An LLM runs on a GPU. It's not gonna call, uh, any code. Uh, so what it does, um, uh, rather is that it generates a JSON representation, so telling you what function to call with what parameter values.

77. 16:56

    And it's gonna map... It's gonna decide which function to call, and it's gonna map the, the natural language sentence and cr- and extract, um, values that it's gonna pass as parameter to the function to be called.

78. 17:12

    And then it's the responsibility of the application code to take that, uh, function, uh, call specification and actually execute the code.

79. 17:21

    'Cause that's what it does.

80. 17:24

    I have a quick question.

81. 17:25

    Yes.

82. 17:25

    How does function call... How does LLM know which functions to use when you're using your custom classification model? Like, do you know under the hood, how does it know?

83. 17:35

    Oh, uh, that is an excellent, um... That is a very good question. Um,

84. 17:42

    well, it's, it's exactly the same, uh, way when you ask a question, for example, when you ask, um, "What is, uh, the color of the sky?" Like very, uh, one of the first use cases.

85. 17:58

    Um, obviously the answer is gonna be blue most of the time. Uh, the way the LLMs, uh, work is that it's a statist-statistical, uh, distribution. Uh, you have the probability, uh, that the, the word that comes after...

86. 18:12

    When you ask a question, what is the color of the sky, the, the, um, the most probable answer is gonna be the color of the sky is, and after is, the most probable color is gonna be blue, uh, because that's how the model has been pre-trained, right?

87. 18:26

    Um, but if you say, "What is the color of the sky?" And in order to get me the, the answer, you onl- you can only use, um, uh, for example, color equal and the value of the color.

88. 18:44

    Then you give an instruction. You tell the LLM, "Hey, here's how... Here's the output that I want." And then it's gonna generate color equals blue. Uh, just because you constrained, like statistically in the, in the, in the world of all the possibilities, uh, that you can, um, answer, it...

89. 19:02

    You're narrowing down the type of, uh, output that you want to be generated, so color equals blue. Imagine that, um... And, and so you're gonna parse that answer, and you're gonna interpret color equals blue into whatever you want with it.

90. 19:18

    Except in the world of function calling, it's not color equals blue, it's, uh, set, uh, dash co- you have a tool, say, uh, um, execute color, which takes a parameter with the value of the color.

91. 19:29

    And instead of having just one, you have many possible, uh, functions. And same, statistical probabilities. Uh, when you... Imagine you have one question to set the color and another function to, uh, order a pizza.

92. 19:45

    Uh, if you ask what the color of the sky is, it's not gonna generate, uh, order pizza call for the same reason. It's exactly how it works.

93. 19:55

    So how do you know when to use function calling versus creating your own classification model?

94. 20:02

    Okay. Well, because a classification model cannot extract values, entities, uh, out of a context and, uh, like, uh, answer many, many answers, uh, across multiple dimensions at, at the same time.

95. 20:18

    Uh, classification is just, uh, uh, one out of, uh, n possibilities. Uh, so it's a totally, it's an entirely different type of task.

96. 20:29

    Okay, let's move on. Um, okay, so I want you to, uh, on your laptops, uh, to open the following URL. Uh, so I'm sorry, it's not very big. Uh, I'm going to spell it.

97. 20:42

    So, uh, microsoftevents.learnondemand.net. So microsoftevents, all attached, .events...

98. 21:03

    Sorry. Oh my God. .learnondemand.net. Is everybody on the page?

99. 21:21

    Okay. Um, then, uh, you're going to log in. Be very careful. This is very, uh, important. You need to choose, um, Microsoft account. Well, usually you're gonna use, um, Microsoft account.

100. 21:37

     Huh, I thought we had other types of...

101. 21:40

     Okay. So if it's a personal account, you need Microsoft account. If you have a corporate account, like if you have a Mic- an account that's part of your company, uh, which is managed by Entra, you need to select Entra ID.

102. 21:54

     So it's, it's, uh, kind of misleading. Do not select Microsoft account if you use a corporate account. Use Entra ID.

103. 22:08

     Then, um, you're gonna, um, land on that page.

104. 22:18

     That page, uh, you're gonna have a redeem training key, um, link. Click on it, and then you're gonna be asked to enter a training key. Enter the following training key.

105. 22:33

     So hopefully, this is big enough and everybody can see. But let me know.

106. 22:47

     Uh, remember, uh, raise your hands if you have any questions.

107. 22:58

     I think that it needs [REDACTED:username] approval. [laughs]

108. 23:01

     Sorry, what?

109. 23:02

     It said it needs an [REDACTED:username] approval for the-

110. 23:06

     What?

111. 23:06

     For your, the organization. So it may not be possible.

112. 23:13

     Uh, you can use also a personal Microsoft account. Like, uh, you can use an outlook.com. You can use, uh, if you have an Xbox, you can use a, an Ex- an Xbox account.

113. 23:25

     Uh, any M- Microsoft account, uh, or any of the consumer domains out there.

114. 23:34

     So in your case, you might want to try, uh, a personal account.

115. 23:41

     And to be honest, in that case today, it's usually, uh, easier. Like you, you do not need to use a corporate account today. There is absolutely no need.

116. 23:51

     Hey, I just have a question about that first slide we had up. Um, so you were saying an agent can go to-

117. 24:01

     And make a graph.

118. 24:03

     Yeah.

119. 24:03

     Is there any reason you're using agents to that as opposed to just chaining all the workflows and injecting the tools and-

120. 24:11

     Uh, you could totally do that.

121. 24:13

     Yeah.

122. 24:14

     Yeah.

123. 24:14

     I'm just wondering if there's, like, a, a reason why you're using agents for that sort of stuff.

124. 24:20

     Well, like I said, so the, um... We need to go back to the definition of what an agent is. Um, and like I said, the, um... So it's a bit of a overloaded term those days.

125. 24:34

     Uh, it encap-encapsulates, uh, a wide range of definition, including the simplest. Today, we are not in the simplest use cases. We are not in the more complex use cases.

126. 24:44

     For example, at the beginning, I said one of the things that an agent can do is be goal-driven and iterate until it achieves the goal. Today, we are not gonna see that specific thing.

127. 24:57

     We're gonna stop one step be- We're gonna be in the middle in terms of complexity. Sorry. Um, we are gonna be, uh, above the simple completion, um, uh, and we are not gonna, uh, see the looping.

128. 25:14

     Um, we are gonna be, we are gonna be seeing a mix of, um, using tools and code interpreter, uh, with mul-multiple data sources where, um, the information from all those, uh, are mixed together to do reasoning and act on the world.

129. 25:29

     We're gonna see that. And yes, you could do that, uh, using just an LLM locally. Uh, nothing would prevent you from doing that, except today we're using a app, um, uh, service which, uh, manages everything server side.

130. 25:45

     Also, I'm going to show the limits that today I'm going to, uh, give examples to, so that you get a sense and you touch exactly when, uh, the, the, the current, um, architecture hits its limits and when you need to go further and, um, and use, uh, a more orchestrated, um, um,

131. 26:09

     planning and orchestration that, uh, iterates and which, which loops until it reaches the goal. Uh, and I'm gonna g-give an example where you see where it breaks.

132. 26:21

     Yeah, thank you.

133. 26:23

     Um, where were we? Turning key. Uh, okay, after you click on Launch...

134. 26:37

     Building.

135. 26:39

     Building.

136. 26:39

     Oh my God.

137. 26:42

     Yeah.

138. 26:42

     It's not pre-built.

139. 26:43

     It's about eight minutes. Once you click on Launch, it takes eight minutes to launch the lab.

140. 26:53

     Okay. Um, oh my. Um, okay. I thought, I thought they would be pre-built.

141. 27:04

     Um, did everybody click on the Build button?

142. 27:08

     Yeah.

143. 27:09

     Okay. Please do. Uh, so that means we have eight minutes with, to answer questions. [laughs]

144. 27:15

     Yeah. [laughs]

145. 27:17

     So, so we just had one. Thank you. [laughs]

146. 27:20

     Uh, do... Yes, one more.

147. 27:23

     So if an agent is just an LLM running in a loop until it thinks that it's done what you've asked it to do-

148. 27:31

     Yes

149. 27:32

     ... how does it know-

150. 27:34

     When to stop?

151. 27:34

     When to... Yeah. [laughs] How, how does it know what, like, when to kick, kick back out of that recursion?

152. 27:39

     Yeah. Uh, so, so today we are not gonna s- We're, we are not, we are gonna see the limits of not, um... Okay. You have two types of agents, uh, two levels of complexity.

153. 27:52

     Um, an agent does not have to do the looping to be called an agent in the simpler, uh, um, area of the spectrum. Um, in order to, uh, to go a step further and do that, uh, that looping, uh, you need to use something like AutoGen, for example.

154. 28:12

     Um, and, um, those types of agents, you need to define a criteria of done, a definition of done, basically.

155. 28:21

     Okay.

156. 28:21

     Uh, and you're gonna have, uh... And the criteria can be implemented, uh, programmatically, uh, deterministically, or using an LLM, which decides is it done. Like, is the task finished?

157. 28:33

     Uh, have we accomplished the goal? And the workflow, uh, engine is gonna loop until the goal is reached.

158. 28:41

     Um, but it, it is the most tricky thing in my opinion, clearly. That's when you're going to, uh, develop, uh, an agent, uh, figuring out when to stop, um, i-is tricky.

159. 28:54

     For example, um, uh, I often, uh, I've done a couple prototypes with BrowserUS, which is a, a famous, uh, open source, uh, agentic system for browser navigation. Uh, when you ask it to complete a task on the web, it's going to navigate from website to website and read the pages and do actions and, and the evaluation of

160. 29:15

     when the task is done, um, is clearly, uh, not always perfect.

161. 29:23

     Um, yes.

162. 29:25

     I have, uh, two questions. Uh, one is related, one might be related.

163. 29:29

     Sorry.

164. 29:30

     One might be related. [laughs]

165. 29:32

     Yeah.

166. 29:33

     The other one is... Uh, so my first question is, are we going to learn today, like, how to send feedback to the agent, like, in case if the agent, uh, gives, like, incorrect answer or incomplete answers?

167. 29:45

     You mean the looping?

168. 29:46

     Yeah, yeah, the feedback.

169. 29:47

     So, uh, no-

170. 29:49

     Yeah.

171. 29:49

     So not today.

172. 29:51

     Not today.

173. 29:51

     Not today.

174. 29:51

     Uh, the other question is, what's, uh, like, in general, what's the difference between, like, an AI agent-

175. 29:56

     Yes

176. 29:56

     ... and an MCP that's, uh-

177. 29:59

     Oh. Uh-

178. 30:01

     I think, uh-

179. 30:01

     So it's very easy Uh, it's a very easy explanation. So basically, MCP is just a tool, a function. You remember I explained what a function is? So, uh, an LLM is able to do function routing, decide which function to call when you have a question.

180. 30:19

     Um, MCP is just that plus, um, management of the life cycle of the program which completes the, the... Which executes the function.

181. 30:36

     Uh, because normal function tooling, if you just take like O-- uh, uh, GPT or Llama and, um, you do function calling, what the LLM is gonna return is just a JSON telling you the func-- the name of the function and the list of the values for each parameter, nothing more.

182. 30:49

     It's your, your responsibility as a, as an application developer to execute the function.

183. 30:53

     So that's why you need a MCP for that [crosstalking]?

184. 30:57

     N-no, it's not that you need it. It's that, uh, it's one of the existing technologies out there that exist that you can use, uh, as a tool. And the advantage of MCP, uh, at least, uh, as a client-side, um, AI application developer, um, is that the MCP protocol takes care of downloading the executable, the binary, whether it's

185. 31:21

     a Node, Python, or whatever, uh, downl- or a Docker image, um, uh, like pull the executable on your machine and execute it automatically.

186. 31:33

     And it's, uh, it's an overall protocol which comes with, uh, uh... And also it encapsulates the possibility for the tool to declare its, uh, functions. Um, so, uh, from the standpoint of the user, you just have to declare, "Oh, I wanna use, uh, a file system, uh, MCP server," or, "I wanna use a Blender, uh, MCP server,"

187. 31:55

     or whatnot, and that's all you have to do. You, you can select it from a catalog, and it's gonna auto declare, uh, what tools it has, uh, and automatically, uh, um, uh, start it and stop it also when you're done with your, uh, MCP client.

188. 32:08

     It's gonna stop all the MCP servers, uh, and clean up everything.

189. 32:16

     Uh, do-- Is it, uh... [laughs] Where are we with the-

190. 32:20

     I guess [crosstalk].

191. 32:21

     Okay. [crosstalking] Uh, where are we? [crosstalking] Oh my God.

192. 32:30

     Does it give a time? Seven minutes? Oh my God.

193. 32:34

     I've got a question.

194. 32:38

     Did we forget to do something? [laughs] [crosstalking] Yeah. Anyway.

195. 32:47

     I had a question.

196. 32:47

     Yes.

197. 32:48

     Um, I was gonna wait and see kinda how the, the demo played out later. But one of the, um,

198. 32:57

     questions that I often find making agents is the balance between making a fairly general agent that can do lots of things and just kinda giving it the tools, not giving it much direction, versus having to be fairly controlled with it, and you're kinda trading off the, like, autonomy to, you know, a bit of reliability.

199. 33:12

     The one that you set up before, I couldn't quite work out if you've literally just given it the tools and then it can do everything with those tools or if you've been quite controlled with it.

200. 33:22

     So how do you think about the trade-off, and when people are using your tools, do they tend to fall more on one side than the other?

201. 33:28

     Uh, it's a more complex answer that, um, that it looks like. Uh, a more, more complex question, uh, than it looks like, um, because of two things. Um, so when you give instructions, um, you know, it's like...

202. 33:45

     I, I was talking about the, uh, the, the, the space of probabilities. Um, the more vague you are, the more you, you leave, uh, options open, and so the more, um, your agent is gonna be able to do wide area, a wide range of things, but it might get it wrong.

203. 34:01

     The more specific you are, um, the more you're restricting the, um, the things that it's gonna do well, uh, but it's gonna do well more often. Um, and then there is...

204. 34:13

     You didn't really ask this, but I'm assuming that's, uh, on your mind, is how many tools can I, uh, give to my, to my agent? Uh, and also another common question that we often get is, um,

205. 34:27

     should I, um, give all my tools to one agent or should I split tools on multiple agents or should I create, uh, should I only give one tool per agent?

206. 34:41

     Um, and the answer is, uh, same. It's, it's, it's complex, but it's kind of, um... I-imagine, um, same. You need to think in terms of, um, statistical probabilities. Like it's, it's, uh, stochastic.

207. 34:58

     Um, and so when... Imagine you have one agent and you give all the tools. The-- And at ev-every single instant, for any question you ask, the LLM has to decide which one of all those tools to call.

208. 35:12

     The probability that it gets this wrong is higher, right?

209. 35:16

     Um, so the solution to that is to create agents which are more specialized by, like, areas of expertise and do some kind of routing and multi-step selection where, um, instead of having one agent and giving all the tools, you...

210. 35:35

     For example, you have a first, a first round of agent that is going to determine, classify, uh, we were talking about classification earlier, classify the, uh, question and say, "Oh, this is a sales question," or, "This is a, a product question," and then route to an agent which is more specialized for...

211. 35:52

     To answer things about sales or things about, uh, products. Um, and then when the answer comes back, the first agent can say, "Oh, do I need to use another agent?"

212. 36:03

     So it's like an agent is like a multi-tool. So you can imagine like a, a tree of tools, and each tool can be composite or a leaf. Uh, you can see that way.

213. 36:13

     And Autogen allows to build such, um, uh, topologies.

214. 36:19

     Yes.

215. 36:20

     If, If, If each one could be, uh, uh, working with a different data, da-database or a different RAG or...

216. 36:26

     Yeah. Yes.

217. 36:29

     Yeah. And then, and then the, the coordinator or the maestro, uh, uh-

218. 36:33

     Yes

219. 36:33

     ... is that [crosstalking]

220. 36:34

     Yes, but you can also have topologies where, um, the, um, the different agents share memory. 'Cause the thing is, um, sometimes you also have the situation where you have, uh, like, um, a team of agents and they are specialized, and it's good that they are specialized because you don't want to, uh, them to, uh, pick the wrong

221. 36:57

     tool, uh, for the reason I just-

222. 37:00

     Or hallucinate or...

223. 37:00

     Sorry?

224. 37:00

     Or hallucinate or hallucinate.

225. 37:01

     Yeah. Uh, well, hallucinate is...

226. 37:05

     Because each one has its own RAG or-

227. 37:07

     Uh, yeah

228. 37:08

     ... its own specific-

229. 37:10

     But what, what's gonna prevent the hallucination is the grounding. So each one of those agents is gonna be grounded in something. Uh, but something which is... can-- some kind of a grounding, uh, is memory.

230. 37:21

     Like for example, if you have, uh, multiple tools, um, that, uh, answer questions about, um, a consumer, um, and you want, and you want to memorize to remember the preferences of your consumer, of your user.

231. 37:35

     Uh, like for example, what's the name of, uh, of, uh, his or her pet. And you-- but you want each one of your agents potentially to be able to use that information.

232. 37:45

     So what you wanna do is take that memory, connect that memory to each one of those agents. Even if you have multiple agents, they can, each one of them, have access to that shared memory.

233. 37:55

     Yeah.

234. 37:55

     Because you want all of them to be able to access the name of the pet.

235. 37:58

     Yeah.

236. 38:00

     So [chuckles] to be honest, it, it's complex. Like, you have many topologies which make sense depending on your use case.

237. 38:06

     Yeah. That's-

238. 38:08

     Where are we? Uh-

239. 38:10

     How long to get back.

240. 38:11

     Ah, finally. Uh, so where are we? Is every... Ah, one more. Uh, still building. One more.

241. 38:18

     One more.

242. 38:19

     Can, can you please raise your hand when it's still building?

243. 38:24

     I can remind them [inaudible]

244. 38:30

     Trying to get that too, but I tried again and it [inaudible] tried a few times and [inaudible]

245. 38:34

     And how did you try? And you refresh or something else?

246. 38:37

     I think, uh, went back to, uh, the very beginning.

247. 38:40

     Okay. So you can- I, uh, already tried twice.

248. 38:44

     Yeah. So like multiple times.

249. 38:45

     Okay.

250. 38:45

     Each time the wait time is less, and then you [inaudible]

251. 38:52

     Okay. Hopefully it's go- So is it still... No. Is it still building? No, you're good. So sorry, can you raise your hand again for those for which it's still building, for, for whom it's still building?

252. 39:03

     So just one. Okay. So I'm gonna move- It's twenty seconds. Okay. So hopefully it is actually twenty seconds. Yeah. I'm gonna move ahead. Um.

253. 39:16

     I think I'm good now. Sorry. It's good. Oh, cool. Okay. Yeah. So everybody is there. Awesome. So let me move on.

254. 39:25

     And let me see, 'cause I want to reconnect.

255. 39:35

     Four zero four.

256. 39:44

     Uh, I heard four zero four. That's not good.

257. 39:54

     Okay. So I have the same environment as you. Um, so I'm gonna show you, uh, how to make the most of it.

258. 40:02

     Log in as [REDACTED:username].

259. 40:05

     Yes. Uh, no. Sorry. Uh, yeah, I went too fast. I forgot. [laughs] Uh, so when you have the black screen, um, I should have, uh... Okay, let me...

260. 40:16

     Uh, when you have the, when you have the black screen, uh, you, uh, the username should be preselected and should be [REDACTED:username]. So there you can, uh, click on password here, which is going to fill the, the password box, and then you hit Enter.

261. 40:34

     Do not type. Uh, every time you see a T like this, you do not need to type. You just click on it, and it's gonna auto type.

262. 40:44

     Um, is everyo- Is everyone on that screen?

263. 40:49

     Yeah.

264. 40:50

     Yeah?

265. 40:50

     Mm-hmm.

266. 40:50

     Okay. Uh, so we can start. Um, ta-da-ra. Okay.

267. 40:57

     Says we're gonna, uh, first on this. I'm gonna move on to,

268. 41:03

     um... So what I'm gonna do here, what you can do, it's not mandatory, but you can click here and go to Split Windows to get more real estate.

269. 41:14

     So that way here, I'm moving the instructions, uh, away, and here that allows me to have more space.

270. 41:21

     Uh, then I'm going to go to, um,

271. 41:26

     Getting Started. Do we need Getting Started? I don't think we do. No, I think we can move on. Uh, no, we do actually. So, uh, we're gonna move on to Getting Started.

272. 41:35

     Uh, what we are gonna need to do is AZ login here. So I'm going to open it from here, Terminal.

273. 41:46

     Type AZ login. Uh, type select work or school account,

274. 42:02

     but this is very important. Continue. Um, even if you connected the first time, even if you connected with your Microsoft account, um, in this instance now, we are not gonna use your account.

275. 42:19

     We are gonna use a temporarily generated account, which is a work account. So you need to select Work account.

276. 42:28

     And there you're going to, um, select the us... So in the Resources tab here, in the instructions here that I, uh, in the pop-up, you click User. It's gonna auto-type.

277. 42:45

     Then Next. Then the password. Sign in. Uh, yes.

278. 43:08

     You're all set. There we go. Um, okay, so now I'm logged into AZ. So I can go back to the instructions here, and we're gonna have to, uh, type that command, uh, because in Azure you have some roles that needs to be assigned, and that's something we could not

279. 43:33

     automate as part of the lab, uh, provisioning. So that's something you need to, um, execute manually. So you just type that command, paste it in the terminal here. You're gonna have to, uh, say, uh, to agree to the warning and say Pass Anyway.

280. 43:49

     Enter. And you're gonna see a bunch of, uh, logs, and at the end, normally everything should be fine.

281. 43:59

     Okay, here we go. So you have a JSON. Is everybody there?

282. 44:04

     Yeah. Okay. [background chatter] Um, okay, so now finally we're gonna be able to open the workshop. So you, uh, type that command, which starts with Git Clone.

283. 44:19

     So what we're gonna do is we're gonna check out the Git repository from GitHub. Uh, and we're going to build the project and open it and install, uh, some code extension and open it in, um, VS Code.

284. 44:33

     So I'm going, I'm going to go back here, type the command, pass anyway,

285. 44:40

     Enter. So like I said, it's cloning the repository.

286. 44:47

     Creating the Python virtual environment. So today we're gonna use Python for this, uh, workshop.

287. 45:05

     Where do you access the instructions? Excuse me? Where do you access the instructions? Uh, where do you... Oh, um, so when you, uh, you get into the VM, you just open Edge.

288. 45:19

     You open the browser, and the instructions should be displayed, uh,

289. 45:25

     right away. So far from the desktop. Right here? Yeah. Okay.

290. 45:31

     Yes. Um, oh, you must have clicked on a link.

291. 45:49

     Uh, no, maybe not. I don't know. Um... [background chatter]

292. 45:55

     I don't know how you get here. Um,

293. 45:59

     can you go back? Uh. Thing is I don't know how you get here, and I'm not sure what the,

294. 46:15

     uh... Can you see? Yeah, I can start over.

295. 46:24

     Oh, here we go. Um, so I was at this step where, uh, I cloned the repository, and I installed, uh, a PDF extension. Then we're gonna move on, and we're gonna open, uh, uh, vis-- VS Code.

296. 46:49

     There we go. Okay. So now we are with VS Code here.

297. 47:02

     So now, uh, we still have a few setup, uh, steps, uh, yeah, before we can, um, get to coding. So, uh, you're gonna have to go to the Azure AI Foundry here.

298. 47:15

     Sign in. Uh, for the sign in, use the, that user from the, uh, instructions pane.

299. 47:38

     Password. Okay. Stay signed in, yes. Okay. So Azure AI Foundry. So now we are in AI Foundry.

300. 47:56

     Um, then, uh, what we're gonna do is we are going to search for...

301. 48:04

     Da, da, da. Go down, go down. We're gonna... Oh, my God.

302. 48:10

     Come on. [chuckles]Uh, so we're gonna, uh, open the project,

303. 48:16

     uh, here. So be careful. You need to select the project, not the hub.

304. 48:26

     Once you're on the project, you can ignore those pop-ups.

305. 48:36

     Okay. Uh, and we're gonna need that project connection string here. So you can copy it,

306. 48:45

     and we're going to paste it in VS Code. So once you have copied the project connection string,

307. 48:56

     you can go back to, you can go back to VS Code.

308. 49:00

     Uh, search for the env.-- .env.sample file, rename it. So we're gonna remove the sample.

309. 49:14

     And we're gonna paste the connection string here.

310. 49:18

     So that connection string is what allows us to connect to the AI Foundry project in which we're going to deploy the AI agent.

311. 49:27

     Uh, be careful when you paste it to not forget characters, to paste between the double quotes. Uh, otherwise it's not gonna work. Um, so once we've done that,

312. 49:39

     we can move on. So as you can see, we're gonna use GPT 4.0. We can save the file.

313. 49:47

     Okay. Is everybody at this point? This is important because now finally we have configured all the... Oh, I get a no over there. Uh, can you guys go help?

314. 50:00

     Um, uh, Mark, um, is, uh... Is everybody, uh, else at this point? Okay.

315. 50:16

     So do we have everyone at this step?

316. 50:20

     Basically just deciding how everything fits into [background noise].

317. 50:32

     Is e- is everyone set up? Yes? So we can, uh, we can look at the code?

318. 50:40

     Yes.

319. 50:41

     Okay. So the first thing that we are going to, uh, to, to look at first is that I'm going to explain quickly what the files are. So the main file that we are going to look at today is the main.py file, uh, which is the entry point for the application, and it's where everything is configured.

320. 51:00

     The, the goal of this workshop, um, i- is not to give you like, uh, what code should look like in production. The goal of this workshop is for you to understand how it works, to understand all the pieces.

321. 51:14

     Uh, because once you understand, you're gonna be able to use any of those, uh, other frameworks. I mean, this one or another one, it's gonna be the same. What matters is to understand how an LLM works, how function coding works, how grounding works.

322. 51:27

     That's what matters. Um, so and the sales data, uh, yeah, that's where the SQL query generation logic is. Uh, you have some streaming link. I mean, you have a bunch of, of things.

323. 51:42

     Uh, another directory which is very important is the shared/uh, instructions, uh, directory.

324. 51:52

     Okay. So let's, let's move on. So the first step, the most important thing, uh, is to understand function calling. So the first example is, um, uh, is gonna show you what it does.

325. 52:07

     So, um, in sales data, let's look at, uh... Where am I? Yes. Sales data.

326. 52:27

     Uh, so that's where all the, uh, SQLite, uh, logic is. Uh, so you have, um, a bunch of, uh, functions.

327. 52:35

     Yeah, yeah. Yeah.

328. 52:44

     Um... [background noise]

329. 52:54

     .

330. 52:56

     So as you can see, uh, as you can see, uh, here, that function takes a SQL query. I mean, let me scroll so that you can see it. It's a SQL query.

331. 53:08

     So the LLM is gonna be the one generating the SQL query compliant with SQLite syntax and pass it to the function. And the function is actually pretty dumb. The function doesn't, does not do much.

332. 53:21

     It just, um, uses the SQLite, uh, driver and executes the query, nothing more.

333. 53:29

     That's because all the smartness of generating a SQL query is done by the LLM. When you think about it, an LLM is, is very good at language, and SQL query is, is a kind of language like coding, like Python.

334. 53:41

     And so... And it happens that when the large language models are pre-trained, they are pre-trained on the massive amount of, uh, GitHub repository, and a lot of them contain, uh, SQLite queries.

335. 53:55

     So if you use a common, a common database, it's gonna work. Uh, like PostgreSQL, MySQL, um, uh, SQLite, uh, MongoDB. If you use an exotic database that nobody has ever heard of, it's not gonna work, uh, because the, the model needs to have been tr- pre-trained on it.

336. 54:17

     Um, here we go. Okay. So the exercise is, um, in the main.py file. So I'm gonna open the main.py file here.

337. 54:32

     Uh, so we're importing a bunch of packages. Uh, so those are the packages which are required to connect to AI Foundry, uh, to get the models, uh, to authenticate with Azure identity.

338. 54:47

     Um, uh, dot env is the package that allows to load the environment variable from a dot env file that we edited previously. Uh, you have some logging.

339. 54:58

     Okay. Let's mo- let me move on. So here, that's how you, uh, connect to the AI Foundry, uh, workspace.

340. 55:08

     Or project, I should say. Um, okay. And so line fifty-nine, what you're gonna do is comment the first instruction files.

341. 55:19

     Um, and then we're gonna, uh, look at its content. So if you go to instructions here, you open function calling here.

342. 55:31

     Uh, those are the instructions that are going to be passed, uh, to, uh, the agent.

343. 55:39

     Uh, you are a sales, uh, analysis, uh, agent for Contoso, retailer of outdoor camping gear. Uh, so it te- it explains what the, uh, what personality the, uh, agent should have, uh, what the mission of the agent should be, help users by answering sales-related questions, uh, it lists what tools are available.

344. 56:01

     Uh, so here, sales data tool use only the Contoso sales database via the provided tool, uh, with the name of the function. Uh, so, uh, to be honest, those instructions are optional.

345. 56:16

     They are made to, uh... Actually, it echoes with, with the question you were asking earlier. Um, uh, I didn't get your name, but-

346. 56:25

     James.

347. 56:25

     James. Uh, where you were asking how specific you must be. The tool, uh, is- comes with its own schema, and the LLM is gonna understand the JSON schema, uh, with- uh, of the tool, which explains the function name, uh, the parameters, and you have an explanation of what each parameter is.

348. 56:43

     Here in the system, um, um, instructions, you can add some more information that are more specific to your application or to the agent using the tool.

349. 56:54

     Uh, then you have information re- uh, regarding, uh, formatting and localization, um, and examples, et cetera. Uh, well, it's quite, uh, thorough.

350. 57:06

     Um, and, um, here-- So we are specifying the instruction files, and we are also specifying which tools to use. So here we're gonna use the functions, which contains only one, uh, function tool, which is async, which is a function I showed to you earlier.

351. 57:27

     Um, here we go. Uh, and also the documentation here, um, the Python doc is passed to the LLM. So what you write here is important because the LLM is gonna interpret that documentation as well as the name of the parameters, uh, plus the, the documentation of the parameters to, uh,

352. 57:51

     to understand how to call your function. Um, and then I'm going to, um,

353. 58:01

     open a terminal, uh, command palette. And to be honest, I, I never remember where the terminal is. So what I do is that I cheat and, [chuckles] uh, I create a terminal, um, here.

354. 58:21

     There we go. I need to remember we're on Windows. Usually, I have a Mac. Um, okay, okay.

355. 58:35

     Um, okay, we have looked at this. I have explained to you all of this. Run.

356. 58:45

     So now we're gonna hit F５. No. Here. F５. Or I could do that too. Could click on the play button here.

357. 58:56

     Okay. Yeah, you don't even need to open the terminal. You don't even need to do what I just did before. Just, uh, it's a habit I have. But, um, you can just click, uh, run here.

358. 59:06

     It's gonna work. I mean, hopefully. We'll see.

359. 59:11

     So, uh, [laughs] it's doing what I mentioned earlier, which is that first you need to create the agent.

360. 59:20

     Uh, once you created an agent, you're gonna be assigned with an ID because it's stateful. Like, the agent is actually an entity which lives in the AI Foundry, uh, project.

361. 59:31

     Then we're going to enable the auto function call, uh, calls. Uh, we're creating a thread.

362. 59:38

     Um, so the thread is basically the conversation, which is stateful too because, uh, you don't have to save the messages yourself. Uh, and then we can finally enter a query.

363. 59:50

     So I'm going to go back, uh, because we have a list of questions we can ask here. What were the sales by, uh, region? I go back to the terminal and paste it.

364. 1:00:03

     No, sorry. Wrong terminal. This one. Um, and if it works correctly, we'll see.

365. 1:00:15

     Yes, it does work. So to be honest, every time I see that, uh, it, uh, it still amazes me, even if I've done-- I've been doing that for a long time.

366. 1:00:23

     Uh, because, um, basically from the simple question that we ask in plain English, uh, based on the knowledge of the schema of the database, um- Which is somewhere in the code.

367. 1:00:39

     Uh, it can automatically generate such a query. Um, like, let me go back to the question I asked.

368. 1:00:48

     What were the sales by region? So we are asking what the sales are.

369. 1:00:55

     Uh, so where is it from? From sales data, 'cause we are pulling from the sales data, uh, table, and we want to group, um, by region. So we want to sum the revenue,

370. 1:01:15

     group by region, and automatically, it limits to three, and I believe it's because in the instructions, there is an instruction that says that by default, you should limit to three.

371. 1:01:25

     Uh, and so, um... No, three. Sorry. Three, zero. Uh, so that's how we get that, uh, great table. Uh, and we can see that, uh, for Africa, we had, uh, five millio-- five point two million, Asia-Pacific, five point three million, uh, et cetera.

372. 1:01:43

     So we have our table, and now we can ask, uh, follow-up questions.

373. 1:01:49

     What was last quarter's revenue? It gets more tricky.

374. 1:01:56

     To be honest, uh, I, I, I don't know the scheme of the database very well, so even for me, like, I, I would have to look at the... If I had to write a SQL query, I'm sure you've all done that.

375. 1:02:05

     Remember, the goal of such an agent is to enable non-technical people to use technical tools. Um, and in this case, the technical tool is a SQL database. And so here, uh, using, um...

376. 1:02:21

     Oh. Um, ha, don't remember that one. Uh, what's year, um...

377. 1:02:32

     Let's try in twenty-twenty four.

378. 1:02:34

     Yeah. Yeah.

379. 1:02:42

     Okay, so this is interesting. Okay, so we did find, uh... So the question was, what was last quarter's revenue? So I'm assuming that it's the, um... Yeah, select the sum of the revenues, uh, from sales data where year equals twenty-twenty four.

380. 1:03:04

     And we ask for the last quarter. Sorry, did we say last?

381. 1:03:11

     Last quarter. So it selected seven, eight, nine.

382. 1:03:23

     Um, so I'm not sure why it selected, um, those three months

383. 1:03:30

     as opposed to, uh, for example, ten, eleven, twelve. So, uh, let me, uh, uh, correct it, actually. It's an interesting experiment. Every time we do that, obviously, we get different responses, right?

384. 1:03:42

     So, um, actually-

385. 1:03:45

     I think it's because Oro's last training date was November of '24, so it's making last quarter from November is seven, eight, nine. So before November.

386. 1:03:55

     Um, it, it, it's possible, but, um, actually, let's say, uh, we're

387. 1:04:04

     in-

388. 1:04:05

     I got one, two, three, five

389. 1:04:07

     ... April.

390. 1:04:07

     Yeah.

391. 1:04:09

     So it, it works. So I just said, "Actually, we're in April," which means that the last quarter is January, February, March. And as you can see, it changed the request with one, two, three.

392. 1:04:25

     So one thing we could do to improve, uh, this example is add a date tool. We could add a tool that, uh, allows the LLM to ask for what is the current date so that we, uh, would not have to enter it manually.

393. 1:04:41

     Well, I'm gonna leave it, uh, as an exercise.

394. 1:04:46

     Um, which products sell best in Europe? Okay, so this time...

395. 1:05:07

     So this time it is interesting because we are generating a qui-- uh, a query against, against a different dimension, which is, uh, the product type. So as you can see here, we are grouping by product type.

396. 1:05:21

     Okay, I'm gonna, I'm gonna press on because you understand the, the concept. Uh, I'm gonna ask the last one

397. 1:05:29

     'cause I wanna move on to the next examples and see how... what happens when you mix multiple data sources.

398. 1:05:38

     Okay. Same one by region. Let's move on.

399. 1:05:44

     So

400. 1:05:44

     Yes.

401. 1:05:46

     So now we have a function, uh, that knows how to execute the SQL statements, right?

402. 1:05:51

     Yes.

403. 1:05:52

     How did the LLM get about the tables actually?

404. 1:05:56

     Uh, yes. Good question. So let's go in here.

405. 1:06:52

     Okay, so this is how. The way it works is that at initialization of the agent, uh, so we are loading the instructions from the instructions files. Uh, so if I go back to, uh, da, da, da, load, replace.

406. 1:07:09

     If I search for this in the, uh, function calling instruction file.

407. 1:07:17

     Here we go. So sales data tool. Uh, so here, uh, and I passed too fast on this the-- w-when I was going through the file and explaining to it, it to you.

408. 1:07:28

     But so th-what's gonna happen here is that, uh, the instructions of the agents is gonna contain what the schema of the database is. That's how the LLM knows what query to generate and what tables and columns are in the database.

409. 1:07:46

     So, so without actually passing this instruction, um-

410. 1:07:50

     It would not work at all.

411. 1:07:51

     Oh.

412. 1:07:52

     Like, no way.

413. 1:07:53

     Oh.

414. 1:07:54

     'Cause it would have, uh, it would completely hallucinate the, the schema. That-- You're grounding basically the LLM with what the, the schema of database is.

415. 1:08:05

     Oh. Can I be more specific? Like, what if... Just saying, like, what if one column has some confidential data, uh, would you want to,

416. 1:08:13

     uh, get data-

417. 1:08:14

     Yes, you could.

418. 1:08:15

     I can-

419. 1:08:16

     Yeah.

420. 1:08:17

     Okay.

421. 1:08:17

     Yeah. You could, uh... And actually, you don't even have to be too fancy for that. You can literally add it to the instructions file. You can say, "Hey, the column X of table Y is confidential.

422. 1:08:29

     Never return it." Disclaimer, uh, it is not bulletproof. It is not guaranteed that the LLM is gonna follow those instructions. If you have a confidentiality, um, uh, I mean,

423. 1:08:44

     uh, matters, uh, problems of that, uh, in that space where you want to restrict access, uh, you should... Somebody mentioned, uh, IAM or, um, um, I think it was you.

424. 1:08:55

     Um, you should implement this. Uh, you should add column level. And the best is to do what you would do with normal code. Because even if normal code is, uh, non-- is deterministic, and the odds, uh, of a user getting access to something he's not supposed to get access to, it's still possible.

425. 1:09:14

     You can still, uh, use, uh, SQL injection, like all sorts of hacking, uh, to exploit, uh, normal code. Obviously, with, uh, LLMs, you-- it's a whole different area. Uh, but, uh, so you should implement the exact same safety measures you would for a normal, um, program.

426. 1:09:34

     Uh, where were we? Okay. Yes.

427. 1:09:38

     If you were supposing that the, the schema is, is kind of perfect, it's, uh, perfect work. We have schema that, that will point, I mean, our agents to the right, I mean, table to the right-

428. 1:09:49

     I'm not sure what you mean by perfect because the, the schema-

429. 1:09:51

     It's-

430. 1:09:51

     -of a SQL database is, I mean, has to be perfect, I hope. [laughs]

431. 1:09:56

     Yeah, yeah, yeah. I mean, but, but-

432. 1:09:57

     At least matching the data.

433. 1:09:58

     Yeah, but I mean, uh, it de-it depends on the, the, the, the, the one that created, I mean, the-

434. 1:10:05

     Okay

435. 1:10:05

     ... depends on, on the name of the columns. Depends on the meaning-

436. 1:10:08

     Okay, so you mean clear. You mean, uh, interpretable.

437. 1:10:10

     Yes.

438. 1:10:10

     Okay. Yes.

439. 1:10:11

     Yes, that's it.

440. 1:10:12

     Yes.

441. 1:10:12

     So, so, so this, in this case when,

442. 1:10:16

     I mean, a real world-

443. 1:10:17

     Same answer as, uh, regarding security. It's the exact same answer. He, he asked: How can I make sure that confidential information is not returned if I want to? You add an instruction saying, "Do not use that column.

444. 1:10:32

     It is confidential." To answer your question, what you would do is document your schema. It would say that table which has a cryptic name that nobody understands is actually the list of orders.

445. 1:10:44

     Yeah. So it's, it's kind of semant-semantic layer-

446. 1:10:47

     Yeah

447. 1:10:48

     ... in some ways.

448. 1:10:48

     Yes.

449. 1:10:49

     All right. Thank you.

450. 1:10:50

     That's exactly how it would work. Um, okay, let's... [sighs]

451. 1:11:08

     Uh, where am I? I'm lost. Um.

452. 1:11:11

     So [inaudible].

453. 1:11:16

     We did that. Okay. We... Okay. Okay, it's basically explaining what I explained. Okay. Blah, blah, blah. Breakpoint, no.

454. 1:11:25

     Um. Okay, so this is interesting. Let me try that one.

455. 1:11:39

     And I'm gonna increase here. So what regions have the highest, uh, sales?

456. 1:11:50

     Okay, so that's very interesting. Uh, why is it interesting? Because the-- to answer the previous question,

457. 1:12:00

     which was total shipping costs by region, remember that the LLM has access to the context of the discussion, of the past messages. It's stateful. It has all the, all the, the, that context of the answers and questions that were previously asked.

458. 1:12:22

     Uh, and it uses it when you ask a new question. So for example, when I asked total shipping costs by region, it did not have the information required to answer within the previous answers, so it executed the SQL query.

459. 1:12:39

     But now, when I asked what regions have the highest sales, it can actually use the table that was generated previously. There is no need to execute one more SQL query because it has everything it needs.

460. 1:12:54

     Exactly like you-- I mean, without doing anthropomorphy, but, uh, exactly as you, if you were reasoning about the problem, and you were like, "Do I need to go write a SQL query in order to get that information?"

461. 1:13:06

     No, you, you don't. You have it already. So the LLM is gonna do that reasoning.

462. 1:13:14

     Uh, that's why here you don't see any SQL query executed.

463. 1:13:18

     Okay. Yes.

464. 1:13:20

     How do we know it's doing math the right way? 'Cause, like-

465. 1:13:23

     It's not-

466. 1:13:24

     Yeah

467. 1:13:24

     ... doing math the right way. [chuckles] Uh- [laughs] ... but here, we did not ask it to do math. We asked it to re-- uh, to... Oh, you mean because it-- there is a top.

468. 1:13:34

     Okay.

469. 1:13:35

     If, if you're adding two numbers and-

470. 1:13:37

     No. So, so highest... Okay, that's a very interesting question. So highest sales is a-not about doing ma-math, it's about comparing. Like what's bigger or smaller?

471. 1:13:50

     Eleven or one.

472. 1:13:52

     Yeah.

473. 1:13:53

     Or-

474. 1:13:53

     And that's easy because that's something that does not... I mean, easy. Uh,

475. 1:13:59

     what, what I mean is that LLMs are bad at math. They are bad, bad at calculus, bad at addition, multiplication, division. Uh, um, it's gonna get it wrong. Um, uh, and it's funny because I have an anecdote that's from this morning.

476. 1:14:13

     We had, uh, a team meeting [chuckles] and, uh, we were wondering what the compound, uh, uh, value of a monthly... You, you were there. Um, and, um, how... What would be after a year the compound effect of a daily, uh, rate of increase of one percent?

477. 1:14:33

     And in my mind, I remember the formula for this, which is you, you take the percentage, you add one, and you put it to the power of the period, of the number of periods, right?

478. 1:14:42

     And I executed it, and I got a number. Somebody else was like, "Oh, that's big." He typed it in, in GPT, and he got a different answer. And, uh, the LLM got it completely wrong.

479. 1:14:54

     It was, it was erroneous. Uh, it was very convincing. It looked like it was doing wrong, but-- uh, doing right, but it was not. It was, it was... The original answer, which I typed in Google Sheet with, uh, like a formula, was correct.

480. 1:15:07

     So to answer your question, when you need to do math, you need to, uh, uh, use a tool. Here, we're using code interpreter, which can do math, but to be honest, I've tried.

481. 1:15:18

     It's not the best way to do math, to use code interpreter. It's good at generating diagrams, at, uh, reading files, extracting information from a CSV, from, uh, other types of structured documents.

482. 1:15:31

     It's very good at that. If you try to do math with code interpreter, for whatever reason, which I cannot explain, it does not work very well. Uh, when you do-- want to do math, it's better to create your own tool, like calculate.

483. 1:15:42

     Uh, and it takes like a LaTeX expression or some kind of mathematical formalism to represent an expression. You have plenty of mathematical calculators out there, um, that you can use, uh, to, uh, do the actual and accurate, uh, math calculation.

484. 1:16:00

     Uh, and so that the LLM can delegate to a tool the responsibility of doing calculations right, exactly as you as a human would do it.

485. 1:16:12

     Uh, because, uh, me, uh, I don't know, but one percent to the power of three sixty-five, uh, uh, I don't know how to do that, uh, myself. I use a calculator for that.

486. 1:16:22

     Uh, and I'm gonna have to reconnect again. Oh my God, this is annoying.

487. 1:16:33

     Okay. So where was it, was I? Um,

488. 1:16:39

     okay. What were the sales of tents? And here we get a query.

489. 1:16:56

     Uh, we are-- We need to ask again to do a query on the database, uh, because in the past information, we do not have that information. We don't, we don't know the drill down of tents in the United States in April 2022, so it needs to execute a SQL query.

490. 1:17:16

     Okay. I mean, you get the point. Let's move on.

491. 1:17:21

     I'm gonna, uh, stop the tool. Uh, exit. Okay.

492. 1:17:30

     Oh my God, what's happening now? No, I don't want to meet and chat with friends and family right now. [chuckles] [laughs]

493. 1:17:40

     Okay. [laughs] Um, sorry, Mom. Um, okay. Let's move on.

494. 1:17:54

     Um, where am I? Okay, grounding with documents. Very important.

495. 1:18:01

     So we're gonna do RAG. Uh, vector store. You were asking if we're using a vector database for whatever reason. I was blanking, but, uh, yes, we are. Um, and, uh, so we're going to uncomment, uh, those lines, uh, in main, uh, dot py.

496. 1:18:16

     So I'm going to go here, go back up. I think it's back up. Uh, let me, uh, search for... Yeah, it's near the instructions. Da, da, da.

497. 1:18:30

     So file search. I'm going to comment that one.

498. 1:18:36

     I'm going to uncomment the file store here. Can I do that? Yes. Cool. No? Okay. It works just, uh... No. Okay.

499. 1:18:49

     Oh my God. Can I do that? No.

500. 1:18:56

     Sorry, I was trying to, uh, select multiple lines at once, but I'm gonna have to do it, uh,

501. 1:19:03

     over the way. [lip smacks] Okay. So now we are, we're defining-- we're creating a vector store. Uh, I'm not gonna go into the details of the creation.

502. 1:19:18

     Uh, there is a, a utility which, uh, does all the heavy lifting. But basically what it does is that, um, it creates an AI search vector store, store. Uh, it reads, um, all the, um, the documents, uh, which are, uh, here.

503. 1:19:39

     Oh my god, the Wi-Fi today. [clears throat] I'd like to click on this, but... [lip smacks]

504. 1:19:55

     Um. Okay, I'm gonna try to reconnect.

505. 1:20:24

     Okay. Uh, password. [lip smacks] Okay, so like I was saying, uh, I was gonna show you the files. Uh, so... [chuckles]

506. 1:20:39

     Okay. Um, where are my files? Oh, here we go. Um, so it's a PDF

507. 1:20:49

     which, uh, contains all the product information for my products, for tents, for whatever. Um, and so what this function does is that it reads the PDF, chunks it, like, uh, cuts it in small pieces, and uploads all those pieces into, uh, the vector, uh, database.

508. 1:21:06

     I'm not gonna go into the details. I can... If we have time at the end, I can explain to you in, in excruciating details how that works, um, but, uh, not right now.

509. 1:21:16

     Um, so toolsets, yes. Um, I think I wanted to show you something else, but for whatever reason, I'm blanking.

510. 1:21:30

     Uh, I forget. Um, yes, I remember. I wanted to show you the difference between, uh, the instructions file we were using before and this one. So function calling and file search.

511. 1:21:46

     So compare selected. So I'm using the diff tool here included in VS Code

512. 1:21:55

     to, uh, show you the difference. And so here, what's interesting is that you can s- you can see the difference between the instructions. Um, so...

513. 1:22:08

     Okay, I'm gonna... Yeah, that's better. So, uh, in the file search, um, instruction, we have new instructions. So now we have a Contoso product information vector store. Uh, we have a search tool, uh, which allows to search into the, uh, vector database.

514. 1:22:29

     Um, and we have a few different, uh, things when it comes to content and clarification guidelines, uh, such as the kind of questions that you can ask, you know, with new questions like what brand of tents do we sell, which you could n- you could not ask before, uh, because you didn't have the brand information into the

515. 1:22:46

     sales database. Um, that kind, uh, of thing. So what's interesting with the use case here, uh, and really the, the colleague that created that content, it was very, very, um...

516. 1:23:01

     He created something very interesting because it allows to show many things. Uh, like when you have a source of information, a source of data, think about how before when we had to mix, um, different, uh, data store, different databases, and we had to inj-inject them into some kind of big, uh, cube, uh, system to-- OLAP syst- um,

517. 1:23:23

     yeah, OLAP or whatever, I for- even forget, to do some very complex, uh, aggregated queries to link things together. Now you can do that with AI. Uh, like here, when I want to link to join information from a database with information in a PDF which is not structured, I can use the LLM for that, uh, which is

518. 1:23:41

     quite extraordinary. Um, uh, so when I, when I hear all the skeptical-- the skeptics about AI, I just don't understand because, I mean, that's just insane what you can do.

519. 1:23:52

     Uh, anyway, I'm just gonna skip for the rest of the difference between those instructions. Um, but I just wanted to show you that. Now we're going to execute it.

520. 1:24:03

     And I can type run. Okay. So as I said, it's uploading the PDF. Uh, and actually the chunking, I said it was chunking. Actually, the chunking is done by AI search, I think.

521. 1:24:20

     I forget if it's client-side or server-side, but I think it's server-side. It's one of the features of, uh, Agent Service, is that you do not have to, uh, uh, take care of chunking yourself.

522. 1:24:31

     Uh, creating blah, blah, blah. Okay, okay. Okay, same. So now we can, uh, type our query.

523. 1:24:40

     Um, I explained this. Run, run. Okay, ok- oh, here are our questions. So what brands of tents do we sell? Now we can ask this

524. 1:25:02

     Um, da, da, da. And so this time... So what's interesting is that this is not using the SQL database. This is only reading from the PDF, nothing else, because there is no sales information required to answer that question.

525. 1:25:17

     Also, remember that I killed the previous instance of the agent. I created a new one. So all the past conversation was lost, uh, uh, on purpose. This is, um...

526. 1:25:29

     I could have reused the same conversation if I wanted to, to con- to keep the previous context, but I do- I don't want. Um,

527. 1:25:38

     okay. So outdoor living and alpine gear, plus some information about it. Okay, cool. Simple enough. Now, that's interesting. We're asking which one do we sell. So are we gonna query the database?

528. 1:25:59

     No, not yet. Um, oh, you know, okay. That's because we do not sell hiking shoes. So it's not using the SQL database, it's just using the PDF. But what's interesting here is that there is no information about hiking shoes in the PDF.

529. 1:26:13

     Okay, let's move on. So now what's interesting is that it's referring to the previous brands we talked about, outdoor living and alpine gear. So what product type and categories are these brands associated with?

530. 1:26:31

     Same, only PDF. Uh, outdoor living, the tents,

531. 1:26:36

     uh, camping and hiking, both actually. Both do the same. Uh, to be honest, it could have summarized it since both do the same, but it did not. Uh, what were the sales of tents?

532. 1:26:47

     Oh, now it's, it's interesting because now we are asking for specific sales information about a specific year. So now it's gonna need to query the SQL database.

533. 1:26:58

     So we are asking for the sales of tents in twenty twenty-four by product type.

534. 1:27:07

     Include the brands associated with each. So what's very interesting here is that the product type and the total sales, I think, come... Yes, the product type and the revenue come from the SQL database, but the brand does not.

535. 1:27:25

     The brand comes from the PDF. So because in the questions before we, uh, read the mapping between the product type and the brand, that's how it's capable of adding the brand into the table here.

536. 1:27:41

     And that's also... I, I'm not gonna do the demonstration right now. I'm gonna do it at the end. At the end, I'm gonna kill the program, and I'm gonna restart it, and I'm gonna re-ask the exact same question, and you're gonna see the difference in the question.

537. 1:27:56

     Hint, because I'm gonna ask that question without the previous question, and because what I said earlier, remember when I said that an agent can be, uh, goal-oriented and relentlessly, uh, work until it achieves the goal.

538. 1:28:11

     This, in this instance, we have not implemented the loop. So it does not have the planning capability, so it's gonna fall short. It's not gonna be able to say, "Oh, but in order to answer that question, I need to look into the PDF and into the, uh..."

539. 1:28:29

     Well, you might get lucky, to be honest. It might do it out of luck because sometimes it does, because it... Sometimes the planning is simple enough that it doesn't need multiple steps.

540. 1:28:37

     But if you... it gets slightly complicated, and you need, like, multiple, um, step planning, it's gonna fall short.

541. 1:28:45

     Okay, next one. What were the sales of alpine gear in twenty twenty-four by region?

542. 1:29:10

     So very interesting, too, because the database does not contain the brand. So here, what it does is say, "Where product type like..." It automatically generates a like, uh, a matching criteria based on the brand because it knows that the brand does family camping tents.

543. 1:29:33

     Okay, so now we're going to generate charts, and we're going to use a code interpreter for this. Oh, and like I told you, I told you before I do that, I'm going to make an experiment, and I'm going to ask again,

544. 1:29:50

     uh, da, da, da. The last question, actually. The most...

545. 1:29:55

     Yeah, this one. The last question. And this one most likely is gonna fall short.

546. 1:30:04

     So now I'm executing the agent again. Uh,

547. 1:30:10

     yes. Okay, and I'm going to create to ask the last question. Right.

548. 1:30:23

     So here, in theory, it's not gonna work. Yes. Uh,

549. 1:30:31

     what the heck? But it worked. Uh, I guess it figured out that it needed to use the product database.

550. 1:30:48

     Or maybe there is an instruction in, um, the file search that says that when you ask a sales question requiring product information to go read the PDF. It's possible.

551. 1:31:03

     Uh, it's possible. Uh, so, um, yeah, I think that's what happened here.

552. 1:31:10

     Oh, you tried

553. 1:31:11

     Yeah. [background noise]

554. 1:31:21

     Uh, I might have to, uh, quit and launch again.

555. 1:31:24

     I know my result is different than yours. I got-- it gave me alpine and alp-- it doubled up on the second result.

556. 1:31:31

     It, it what?

557. 1:31:32

     It doubled up on the, um... When you did it, it said it was just, it was outdoor, outdoor living for backpacking with them camping tents was alpine gear. But, um, for my, it said alpine for the remote.

558. 1:31:45

     Okay.

559. 1:31:46

     This one for one and the other for the other.

560. 1:31:48

     Okay. Interesting. Uh, yeah, it's not, it's not deterministic, so it's not perfect. Uh, it's gonna make a mistake for sure.

561. 1:32:00

     Which is, is that the whole eval thing? Like, I guess that's where I struggle sometimes-

562. 1:32:04

     Yes

563. 1:32:05

     ... 'cause it is that whole-

564. 1:32:05

     So I, I give a breakout session, by the way, on evals-

565. 1:32:08

     Okay

566. 1:32:08

     ... uh, tomorrow.

567. 1:32:10

     Okay.

568. 1:32:10

     At twelve thirty, I think, or two PM. I forget. Uh, but I'm gonna, um, I'm gonna give a break, uh, breakout session tomorrow specifically on how to evaluate agents.

569. 1:32:21

     Um, because yes, it's [laughs].

570. 1:32:24

     It's a little crazy because it's like, "Well, the answer might be right."

571. 1:32:26

     It's a problem. [laughs] It's hard. Uh, okay. Let's move on.

572. 1:32:34

     Code interpreter. Uh, so now, um, we're gonna go back to our main.py, and I'm going to comment that one.

573. 1:32:48

     Okay. We're going to... And every time we keep the previous tools, we just add new tools.

574. 1:32:56

     Okay.

575. 1:32:59

     So we can also just have all the tools running, right?

576. 1:33:02

     Yes. At the end, when I'm gonna uncomment everything, uh, all the tools will be working at the same time. But what's very interesting is to uncomment the... It's a workshop.

577. 1:33:12

     Uh, it is, is so that you get an understanding of exactly the subtlety of how everything is-- works.

578. 1:33:24

     Um, um, okay. So run. So we're gonna look at the, uh, instruction, and we're gonna compare code interpreter with, um...

579. 1:33:48

     If I search compare. So here, with, uh, code interpreter, we add visualizations. We add a chapter, a section in the instruction which explains w- how to do visualization.

580. 1:34:08

     So when you have questions involving visualization, we basically say, "Hey, go use the code interpreter to generate charts."

581. 1:34:17

     Um, and at the end, yeah, PNG. Okay, let's move on.

582. 1:34:30

     Main. There we go. Um, okay. So show sales by region as a pie chart.

583. 1:34:49

     Remember in the previous, uh, in the slide I showed before, we were first asking for the sales by region and then for a pie chart. Here, we're asking for both at the same time.

584. 1:35:01

     Um, and so it's tricky because, like, sometimes, uh, when you need... Because basically what, what this is doing is that it's calling two tools.

585. 1:35:13

     Uh, and sometimes it's simple enough, the reasoning capabilities of the, of the LLM is capable in, like, one, uh, uh, question and answer to, um, uh, it has, is enough reasoning power to say, "Oh, I need to, to call that tool and then this one."

586. 1:35:30

     And sometimes it falls short, and it does not, and it does not have enough reasoning power. And you need to, uh, to use a, a multi-step, um, um, system.

587. 1:35:43

     So here, we, uh, saved the results here, so I'm going to... Well, okay. So here is a diagram, uh, the pie chart diagram, uh, with our answer. I can try to zoom to show.

588. 1:35:56

     Okay, so we have the drawdown of, uh, revenue by region.

589. 1:36:00

     Okay, cool. I go fast on this, but this is incredible. I mean, under the hood, it's literally generating Python code to generate the, the, the pie charts. Uh, so basically what that means is that you can generate

590. 1:36:16

     pretty crazy, uh, diagrams idea. You can, uh, you, you can be pretty imaginative in the type of, uh, information that you want to display.

591. 1:36:26

     Um, for, for the, for the anecdote, uh, last year at this conference, I gave a talk, uh, on code interpreter because at the time it was a hot topic.

592. 1:36:37

     It was brand new. And what I had done, uh, I, I do kitesurfing, and I had extracted, uh, one of my sessions, uh, where, you know, you're on the water and you do attacks, you do, you do turns, you do jumps.

593. 1:36:50

     And I had the XML file of my session recorded from my watch exported. I imported it into a code interpreter, and I asked to calculate how many times I turned or how many times I jumped.

594. 1:37:05

     And from... It was able to read the XML, because XML is a very rich, uh- A strict format, s-self-sufficient, and extracted all the information and the sta- and the sta-- did all the, um, the, the structure and generated the Python code to go through all the i-- all the data points in the file and calculate how many

595. 1:37:28

     turns and jumps and height I, I had during my session. It's a, it's a pretty incredible tool when used right.

596. 1:37:35

     Anyway, end of the anecdote. Um, show s- blah, blah, blah. Okay, um, next.

597. 1:37:45

     Uh, why... Download JSON. Uh, I forget what this one does, to be honest.

598. 1:38:07

     Uh, yeah, okay. I mean, it can be useful, but let's move on. Uh, continue asking questions about... Yeah, because there was no tools involved. Yeah, it just, uh, interpreted whatever was in the context before and download it as JSON.

599. 1:38:23

     Um, uh, blah, blah, blah. Uh, what would be the... Huh, interesting.

600. 1:38:32

     I mean, it's really crazy what you can do, right? You can really, really be imaginative.

601. 1:38:42

     By the way, you hear a lot about, uh, one of the latest, uh, things in AI is deep, deep research. Uh, under the hood, deep research is an agent that has those tools,

602. 1:38:55

     just that it's been, uh, developed by, uh, an army of engineers and, uh, they've made sure that everything works, uh, well. But it's basically what it's doing. Uh, it-- I don't know how it's implemented, but I'm assuming it's a, a mix of AutoGen or Semantic Kernel or some kind of orchestration, uh, system that mixes tools together, uh,

603. 1:39:14

     and it loops until it achieves a goal, and it can run for hours. Anyway. Um, so

604. 1:39:22

     what would be the impact of a shock event, twenty percent sales drop in one region?

605. 1:39:29

     We got it.

606. 1:39:29

     Oh, yeah.

607. 1:39:30

     Oh.

608. 1:39:30

     Can you elaborate on, like, the difference between what this can do, how it's, like, not moving, but, like, if you use AutoGen and you can, like, move through the different tools that it needs and-

609. 1:39:42

     Yeah

610. 1:39:42

     ... have planning orchestration. Like, I see that it has planning and orchestration. It's able to use several tools at the same time.

611. 1:39:49

     Two.

612. 1:39:50

     Yeah, two. Fine.

613. 1:39:52

     Uh, and-

614. 1:39:53

     So, like, what-

615. 1:39:54

     Like, for example, let me, let me try to find an example and, um... Like, for example, if you ask a question, you have two da-data sources, right? One is product, one is sales.

616. 1:40:05

     Uh, imagine you ask a question for which it is obvious that you need to query one, but it's not obvious that you need to query the other. The information is in the other, but you have no hint

617. 1:40:19

     in the phrasing of the question that you should be querying the second in order to, uh, answer the question as a whole. It would fall short.

618. 1:40:31

     This, this doesn't fall short.

619. 1:40:31

     Yes.

620. 1:40:33

     Not the AutoGen one.

621. 1:40:33

     Depending on how you configure it. And this one will... The falling short would also depend on how, what instructions you gave it, because you can hack your way through those kinds of problems.

622. 1:40:44

     Because if you, you can just, in your instruction, add some instruction saying, "Hey, if the user ask for that kind of question, you should also go look into the data source."

623. 1:40:55

     But the, the difference... Sorry, look at. The difference with a, um, a multi-agent, multi-step planning system, uh, with a more complex topology is that you have an LLM which every time you get an answer, looks at the answer and ask itself the question, "Is the answer correct?"

624. 1:41:13

     Um, "Is this answer answering the question or not?" And it's capable of extrapolating. It can, it can, it can do something such as, "Huh, this is not answering the question.

625. 1:41:23

     What do I have at my disposal? What could I do more to try to answer the question?" Then maybe the first, um, step, it did not query the second data store, but it's gonna say, "Actually, maybe we should go look into the data store because actually maybe the answer is in there."

626. 1:41:39

     Or we should, uh, query the internet. Uh, see what I mean?

627. 1:41:43

     And AutoGen can do that.

628. 1:41:44

     Yes.

629. 1:41:45

     Like, if you configure it right.

630. 1:41:46

     Yes. Or Semantic Kernel or other, uh, more, like, collaborative multi-agent system. Yes. But this is... Today, it's an introductory, so we, we don't, we don't go into those more complex.

631. 1:41:58

     It's a, it's a, it's a workshop of its own,

632. 1:42:04

     just looking at, uh, multi-agents systems. Yes.

633. 1:42:12

     So you said that sometimes, I mean, uh, the, the answer is not obvious or, or the questions, the prompt itself, it doesn't point to the right maybe, uh, set of data or the-

634. 1:42:22

     Yes

635. 1:42:22

     ... tool. Uh, so, so, so here, here, what, what about like having on top of those specialized agent a coordinator that works with, with reasoning, uh-

636. 1:42:32

     That's exactly what we were saying. That's a multi-agent system.

637. 1:42:36

     And a coordinator, when you s-use the word coordinator, it's actually one of the patterns. It's a... We call it a topology. It's one of the topologies of multi-agent systems.

638. 1:42:48

     And you can build such a topology with AutoGen.

639. 1:42:52

     Yeah.

640. 1:42:53

     Heads up, it's not trivial.

641. 1:42:54

     It's not.

642. 1:42:55

     Not trivial. The hardest question in multi-agent systems is, uh, in my opinion, how to defi- how to, to, uh, specify the definition of done.

643. 1:43:16

     Um, so we did not open, did we? No, we did not. Okay. So this is

644. 1:43:26

     a simulation impact of a twenty percent sales drop in North America on global sales distribution.

645. 1:43:38

     Percentage of global sales. Oh yeah, because in North America here it's dropping, so the percentage obviously of North America goes down, while the percentage of others goes up.

646. 1:43:58

     Uh, that's a pretty cool example. Uh, what if the shock event was fifty percent? I, I mean, I'm gonna skip because y- you, um, it's pretty ob- obvious what it's gonna do.

647. 1:44:09

     Uh, which regions have sales above or below their range?

648. 1:44:30

     Ah, [chuckles] interesting. So maybe the previous questions, previous question was...

649. 1:44:41

     No, I don't think so. I think it's, um...

650. 1:44:51

     Huh. Do we have a problem? Connectivity problem maybe?

651. 1:44:59

     Let me launch it again.

652. 1:45:32

     Okay. Let me just, um... So we are...

653. 1:45:37

     Okay. I wanna show you real quick the Bing, uh, grounding with Bing search. So I'm gonna skip over that one

654. 1:45:45

     'cause this is important. Uh, very often you want to add, uh, internet search capabilities, so I'm going to comment, uncomment this,

655. 1:45:56

     use, uh, the other instruction file. Ah. What to do?

656. 1:46:10

     Okay, here we go. And the last one, code interpreter multilingual, uh, we skip it. Um, it's important, uh, because that's how to configure code interpreter once you wanna use...

657. 1:46:25

     you want to work on non-English languages, so with specific encoding, specific, um, fonts, that kind of thing.

658. 1:46:35

     But we're gonna skip it for today. Uh, resource not found.

659. 1:46:40

     Huh. Well, that's interesting. Did you have that pro... Did you try to that point? Did you have that issue?

660. 1:46:53

     No. Is it working for you? I got that error. You have the same? Yeah.

661. 1:47:03

     Huh, okay. Um, is it because of the Bing grounding API? Maybe we've had a change. [chuckles] Okay. And I, I'm not sure what happened. Uh, we... it may be 'cause Build was recently, and a lot of APIs have changed, and maybe that one changed too.

662. 1:47:22

     I'm not sure. Anyway, what would have happened is that, um, let's look at the question. Um,

663. 1:47:42

     okay. So here, what beginner tenants do we sell, uh, queries our products, uh, PDF. What beginner tenants do our competitors sell, include prices, needs to go query the internet.

664. 1:47:58

     You need to go, um, uh, query, uh, information about what competitors are out there and what they sell and for how much.

665. 1:48:08

     Uh, so that would use the Bing, uh, grounding, uh, tool, uh, et cetera. So

666. 1:48:15

     same logic as before. Uh, we show, we, we showed, um... And I'm going to, uh, wrap up and go back to my slide.

667. 1:48:33

     Um. But just to finish my sentence, so, uh, what, uh, with bring, bring, bring, uh, one, two, three, Bing grounding, it's just one more tool, uh, that, uh, same as the, um, um, database, uh, query tool, uh, the vector database, uh, file indexing tool, is one more source of

668. 1:48:58

     data that the agent can use, uh, to, um, to, uh, to make informed, uh, decisions, uh, or rather reports. Um, so we've seen how to do more with, uh, LLM, with function calling, uh, like with, uh, a bunch of very powerful tools, um, and, uh, yeah, Azure AI Agent Service.

669. 1:49:22

     Um, so yeah. Um, the UI today was very simple. Uh, it's always a question when you do workshop like this, like do we go with a web UI, which is complicated, and add some React and web stuff that some people might not be familiar w- uh, with.

670. 1:49:36

     That one is like barebone. Like, it's just CLI, the bare minimum of code, focusing, uh, explicitly on making sure you understand how to build such an application and how an LLM works, uh, because it's a whole new paradigm.

671. 1:49:51

     Uh, and, uh, and building, building those is not trivial. And as you mentioned, there is a question of evaluation. And so, uh, when it comes to evaluating normal, just an LLM, uh, question answer, uh, there has been many, uh, frameworks out there for some time now.

672. 1:50:09

     Uh, but r- um, when it comes to evaluating agents, it's a whole new world [laughs] again, because not only you are evaluating one answer, but you need to evaluate an whole conversation.

673. 1:50:22

     Plus, because you need to call tools. So you need to evaluate whether the good tools, the accuracy of which tools were selected. Anyway, we're gonna see all of this tomorrow during, uh, the breakout session, where I'm going to introduce the, uh, Azure AI Evaluation SDK with the agent ca- uh, evaluation capabilities, uh, which is, uh, fascinating.

674. 1:50:45

     Um, you have a, on that QR code, a bunch of, uh, additional, uh, resources.

675. 1:50:56

     And, uh, I'm gonna point back to, um, my contact if you want to-

676. 1:51:03

     One more thing.

677. 1:51:03

     Oh, sorry. There we go. Do, uh, do you have questions?

678. 1:51:19

     'Cause we have three minutes to be- Oh, no. One minute, sorry.

679. 1:51:25

     But, um-

680. 1:51:26

     Oh, sorry. First-

681. 1:51:26

     The tools which, which are used by agent, should they be part of the Azure ecosystem?

682. 1:51:33

     No.

683. 1:51:34

     No.

684. 1:51:34

     Uh, so you can mix. Uh, when you create, uh, in AI Foundry, uh, an agent, you have a bunch of pre-made, uh, tools that are readily available. You just have to click on it and configure it.

685. 1:51:46

     Uh, like I said, uh, I can't remember from the top of my mind. You would have to, to, to go look, to be honest. Uh, but the tools we defined today, they are client side.

686. 1:51:55

     Yeah, we define them, we define them using code. Uh, well, except, uh, Bing grounding, because Bing grounding is, you declare it client side, but

687. 1:52:06

     the actual Bing grounding call is happening on the server side. But the SQL query is client side. So we, we, we had a mix actually today.

688. 1:52:18

     So I know that the Agent Service in AI Foundry is free right now. So what are the... Why should we use the Agent Service versus just the, the serving service for like GPT-4.0, 4.1?

689. 1:52:31

     'Cause I know the Agent Service is limited to 4.0, 4.0 mini, 3.5. Um, so I guess if we can instantiate the tools in the application itself, do we need to use the Agent Service then inside of Azure or can we just stick with the regular old LLM endpoints?

690. 1:52:50

     You, you c- You, you do not need to.

691. 1:52:53

     Okay.

692. 1:52:53

     Uh, like I, like I said, uh, earlier, uh, everything I just showed today with a- agent, Agent... Azure AI Agent Service is just

693. 1:53:04

     a managed-

694. 1:53:05

     Okay

695. 1:53:06

     ... uh, AI system. Managed.

696. 1:53:09

     Okay.

697. 1:53:09

     In the sense that it takes... Like most developers, every time you build an AI application those days, you need a vector database, you need, uh, conversations, you need, uh, to remember those, you need to store tho- store those.

698. 1:53:21

     Uh, you need, um, to search on the web. I mean, those are the, the basic features that every single AI application out there needs. I mean, most of them.

699. 1:53:30

     So Agent, um, Azure AI Agent Service makes it super easy, manages everything for you. Uh, but no, you do not have to use it.

700. 1:53:40

     Okay.

701. 1:53:40

     Uh, you can use, uh, the bare bone, uh, LLM, uh, completion from, uh, two years ago, and you can do whatever you want with it. It-

702. 1:53:50

     Yeah.

703. 1:53:50

     Uh, it's just, it's easier. Uh, you have a full-blown API. It manages persistence. Uh, it's just easier. [outro music]