The A to Z of Building AI Agents

Talk transcript

1. 0:00

   [on hold music] Hello, everyone, and welcome to this workshop I like to call The A to Z of Building AI Agents.

2. 0:21

   So during the workshop today, uh, we'll spend about twenty to thirty minutes talking about the basic concepts of what AI agents are, when to use them, the different components of agents, and concepts that you'll find helpful during the hands-on portions of the workshop.

3. 0:37

   And then you will spend the rest of the time building an AI agent of your own with help and assistance from me. And I have my awesome team back there.

4. 0:46

   There's Tom, Ben, and Fabian. So if you run into issues, call upon one of us, and we'll figure it out.

5. 0:54

   Here's a little bit about me. I'm Apoorva, and I'll be your, uh, lead instructor for today. Five months ago, I stepped into my first ever developer advocacy role at MongoDB, and prior to that, I spent about six years as a data scientist in the cybersecurity space, applying machine learning to problems like phishing detection, malware and ransomware detection,

6. 1:15

   that kind of stuff. Outside of work, uh, I read a lot, try to yoga kinda regularly, and I'm always on a mission to hit as many local coffee shops as I can.

7. 1:27

   A few ground rules before we begin. No stupid questions here today. We are all here to learn, so ask as many questions as you'd like. We'll go over key concepts before getting into the hands-on labs.

8. 1:39

   So during these exercises, we definitely encourage you to, uh, form groups and work together, uh, where you can.

9. 1:47

   Here's a link to the slides and also the hands-on lab that you'll be working through today, and I'll leave this here for a few minutes for y'all to scan.

10. 1:56

    Um, the link and QR code should also be on these like postcards that were just handed out. And if you didn't receive one, then raise your hand, and we'll get you one.

11. 2:07

    Anyone need a postcard? Okay, I see some hands there, here.

12. 2:15

    Tom, right here.

13. 2:22

    Or is it just a collectible gauge? Yeah. They're not collectible. [laughs] [laughs]

14. 2:31

    All right. Moving on [laughs]. So the goal of the workshop is to introduce you to the basic concepts of AI agents and also get hands-on experience with building an agent end to end.

15. 2:43

    So yeah, I'm gonna start off by talking about what agents are, some... what are the AI agent use cases, components of an agent, and then we'll build an AI research agent together.

16. 2:54

    And depending on how long it takes us, we'll-- we may or may not have time for Q&A, but I'll be around to answer questions later.

17. 3:03

    So let's talk about-- start with talking about what are AI agents.

18. 3:09

    So an AI agent is a system that uses a large language model, or LLM, to reason through a problem, create a plan to solve the problem, and also execute the plan with the help of a set of tools.

19. 3:22

    So let's see how agents are different from other techniques for interacting with LLMs, because this will kind of help us build an intuition for when to use agents. So let's take the example of simple prompting, where you simply prompt an LLM to generate an answer based on its pre-trained parametric knowledge.

20. 3:40

    So as you can imagine, this is good for point-in-time general knowledge kinda questions, but probably not too much more, right? Because even if you manage to prompt, uh, the LLM to f-perform really complex tasks, then it might not have the means or information to execute on the task.

21. 3:56

    The LLM in this situation also can't self-revise and refine responses based on either previous or new information, and it definitely doesn't have a means to learn preferences and provide, uh, personalized responses over time, which sometimes is a requirement.

22. 4:13

    Moving on to retrieval-augmented generation, AKA RAG. Uh, with RAG, you can broaden the scope of the LLM by augmenting its knowledge with information retrieved from a knowledge base. So that way you can be somewhat confident that the LLM at least has information required to, uh, perform tasks that you want it to perform, but it doesn't quite solve

23. 4:34

    for some of these other requirements, such as handling complex tasks, self-refinement, or personalization.

24. 4:42

    Coming to agents. With agents, you can give the LLM access to external tools and past interactions which act as the memory of the agent, and then you can prompt it to go through multiple iterations of reasoning and action taking to finally arrive at the final answer.

25. 4:59

    So tools is how agents are able to execute on complex multi-step tasks, and LLMs can also be prompted to incorporate the feedback or output from tools into the reasoning process to, say, repeat steps if necessary or call additional tools as follow-up tasks.

26. 5:16

    Coming to past interactions. Past interactions can be persisted and updated, which means the LLM can now learn from these to provide personalized responses over a period of time. So as you can imagine, tools, memory, and iterative prompts can solve a lot of problems, but there's obviously some known challenges at the moment, such as long-term planning, where the

27. 5:37

    agent i-is expected to, uh, execute complex tasks based on information, a lot of information or information it's learnt over, uh, a long-ish period of time. There's also a high, uh, cost and latency associated with agents because they typically trade these for a shot at higher accuracy.

28. 5:55

    But despite all of these challenges, I think we can agree that agents is how we get the most out of large language models as of today.

29. 6:04

    So let's take some example tasks or questions and try to answer whether or not, um, the task really requires an AI agent. So this one, for example, like who was the first president of United States?

30. 6:17

    Does it require an AI agent to complete this task?

31. 6:22

    I see some people nodding yes [laughs] mostly nos. But I would say no, because the information required to answer this question is very likely present in the parametric knowledge of most LLMs that we know today.

32. 6:35

    So I don't think it requires, um, an AI agent.

33. 6:40

    How about this one? What's the travel reimbursement policy for my company, MongoDB, or your company? Uh, do you think this task requires an AI agent?

34. 6:51

    Two-step process.

35. 6:51

    What's that?

36. 6:53

    Two-step process.

37. 6:53

    Two-step process?

38. 6:54

    Yes, it-

39. 6:54

    What, what's the two steps?

40. 6:56

    It first you need to disambiguate, uh, for this ask because the policy will depend on the location of the employee.

41. 7:05

    That's a good point. Do-

42. 7:07

    I, I did it, so that's how it works.

43. 7:09

    Okay. [laughs] All right. All right. Yeah. So I would say it's a pretty straightforward task, provided the LLM has the r- access to the right information. So to me, it sounds like a better fit for retrieval-augmented generation where the LLM has knowledge, um, access to the right knowledge base than, uh, something complex like an AI agent.

44. 7:31

    How about this guy? Like, "How has the trend in the average daily calorie intake," it's already too long, but, "among adults changed over the last decade, and what impact might it have on obesity rates?

45. 7:41

    Additionally, can you provide a graphical representation of the trend?" Do we think this requires an AI agent?

46. 7:51

    I would think so. Like, I think this task looks like it involves multiple sub-tasks such as at least data aggregation, visualization, and also reasoning through, uh, the results that it's obtained from these various tasks.

47. 8:04

    So I think it sounds like a good fit for agents.

48. 8:07

    How about this one? Uh, "Creating a personalized learning assistant that can adjust its language, examples, and methods based on the student's responses."

49. 8:19

    I see some nods, and I agree. I think this is another example of a complex task which requires also long-term personalization. So again, I think it's a good use case for agents.

50. 8:31

    So the TLDR is use agents for complex multi-step tasks that require integration of multiple capabilities such as question answering, task execution, analysis, that kind of thing, uh, and using all of these to arrive at a final answer or outcome, and also if there is a need for personalization or adapted responses.

51. 8:53

    So as we saw, memory, tools, and being able to reason is what really makes AI agents so powerful. So let's dig a little bit deeper into each of these components, starting with planning and reasoning.

52. 9:06

    So the simplest way to imbue planning and reasoning capabilities into agents is via, uh, believe it or not, user prompts. You can start super simple by prompting the agent to create a plan of action based on its initial understanding of the problem, and this is what we call planning without feedback, since the agent does not modify its

53. 9:25

    execution plan based on any new in- information that it's gathering from tools that it's executing. It's just in the beginning, it creates an execution plan and runs with it.

54. 9:36

    So common design patterns for this kind of planning are chain-of-thought and tree-of-thoughts. Then there's planning with feedback, where you can prompt the agent to adjust and refine its responses based on tool outcomes or even asking it to, uh, critique and reflect upon its own responses.

55. 9:53

    And common design patterns in this regard are ReAct and reflection, and we'll experiment with some of these in today's workshop.

56. 10:02

    So let's first understand chain-of-thought. So chain-of-thought is as simple as prompting an LLM to think through a problem step by step instead of directly providing an answer. Uh, you can do this either in a zero-shot manner by literally saying, "Hey, let's think step by step," or in a few-shot manner where you show it how to work through

57. 10:22

    a complex problem using one or more examples.

58. 10:27

    Then we have tree-of-thoughts, which takes the idea of chain-of-thought up a notch. So tree-of-thought allows LLMs to perform like deliberate decision-making by considering multiple different reasoning paths and having it self-evaluate choices to decide the next course of action.

59. 10:44

    And so it kind of combines this LM, LLM's ability to generate and evaluate thoughts with search algorithms because it can also look ahead and backtrack when necessary to make kind of global choices.

60. 10:59

    Then we have patterns for reasoning with feedback, starting with ReAct. Uh, so what we do here is we prompt LLMs to generate verbal reasoning traces and also co- tell us the actions that it'll take to solve a particular problem.

61. 11:12

    So after each action, we ask the LLM to make an observation based on information or feedback obtained from the previous action and plan what action to take next. And then this kind of, uh, process continues until the LLM or you can intervene and say that you've reached the final answer, so exit the loop.

62. 11:32

    So in this example here, uh, as you can see, the first thing that the LLM does is generates a thought saying like, "Okay, this is how, uh, I need to solve this problem."

63. 11:40

    Then the second is an action step where, in this case, it's determined that it needs to, um, call the search tool with arguments that it's determined, and then it makes an observation saying, "Okay," like, "I don't think I, I have an answer next.

64. 11:54

    This is what I'm going to do next," and does that till it, uh, reaches the final answer.

65. 12:01

    Another technique for incorporating feedback into the planning process is via reflection. And this involves prompting LLMs to reflect on and critique past actions to decide what, uh, action to take next.

66. 12:14

    And you can either, uh, prompt the same LLM to generate and critique, you can use different LLMs or even use multiple agents where one agent generates responses and the other critiques them.

67. 12:27

    But yeah, whatever the architecture, the goal is to, uh, run the generation reflection loop several times before, um, the LLM arrives at a final answer. So essentially trading compute for a better shot at accuracy.

68. 12:43

    The next component we want to talk about is memory.

69. 12:47

    And this component allows AI agents to store and recall past conversations and enables them to learn from these interactions. And as you can imagine, memory is a pretty, um, complex and nebulous concept, but-- And you could break it down into several categories.

70. 13:01

    But broadly, uh, when I think of memory, it's two main, uh, types of memory, much like us humans, right? Short-term and long-term memory. So short-term memory in, uh, the case of agents deals with storing and retrieving information from a single conversation.

71. 13:17

    And long-term memory deals with storing, updating, and retrieving information from multiple conversations had over a period of time. And this is what really helps agents personalize their responses over a long-ish period of time.

72. 13:32

    So short-term memory is relatively easy to implement. Like, how hard can it be to store a single conversation, right? Like, in most cases, not that hard. But unless the conversation gets too long, in which case you need to now start considering, uh, how to condense that list so you aren't overwhelming the LLM with too much information.

73. 13:51

    And some solutions to th- to that are things like retrieving the n most recent messages or summarizing the conversation at the cost of some information loss.

74. 14:03

    Long-term memory, on the other hand, is a largely unexplored area so far since it's non-trivial to decide and implement, uh, what states to track, uh, and how to track them and when to update them.

75. 14:16

    So but I think some patterns are emerging in the sense that the best way to go about implementing long-term memory is to design application-specific agents. That way you're able to narrow down the number of states you want to track and just focus on those, uh, and figure out how to update them.

76. 14:35

    And finally, we have tools. So tools are interfaces for agents to interact with the external world in order to achieve their objectives. And these can range from simple APIs such as search, weather APIs to, uh, complex things like vector stores or even specialized machine learning or deep learning models.

77. 14:57

    So tools for LLMs are typically defined as functions, and most recent LLMs have been trained to identify when a function should be called, and they'll respond with a function signature that you can then use to call a particular function in your code.

78. 15:12

    And tools like LangChain handle the function calling for you, but the basic concept still remains. And to help the LLM identify which function to use, you typically use a descriptive tool name, uh, specify which function to call, provide a pretty detailed description of what exactly the function does, and also, uh, the types of arguments would also be

79. 15:33

    helpful. So finally, the fun part. Uh, you're not here to listen to me ramble on about agents. So, uh, in today's workshop, we'll be building an AI research agent.

80. 15:46

    And the agent's primary objective is to provide research assistance by supplying a list of papers to read, uh, summarizing research papers, and answering questions about research topics.

81. 15:59

    And this is kind of how the workflow of our agent is going to look like. We will use a free and open source model from Fireworks called FireFunction v1.

82. 16:08

    They had just released a v2, but I had prepared my workshop until then. So today we'll use v1, uh, as the brain of our agent. We'll also try out some of the reasoning design patterns that we were just talking about, like chain-of-thought and ReAct.

83. 16:23

    We'll also give the agent access to three tools, one for sub-- uh, getting paper summaries, uh, one for getting, um, a list of papers to read, and the third one being answering tools using, uh, a MongoDB knowledge base.

84. 16:38

    And finally, we'll also explore adding short-term memory to the agent and persisting it to, uh, a database in MongoDB.

85. 16:48

    But yeah, uh, very qu- soon we are going to break for our first hands-on portion, but just some things to keep in mind. Uh, each time we, each time we break for a hands-on section, you'll navigate to the hands-on lab at the QR code that you have at your tables or you just scanned, and you'll work through

86. 17:05

    one or more sections at a time. And you'll see these emojis sprinkled all over the place. So this like open hands emoji and the superhero emoji indicate hands-on sections, except, uh, I would highly advise do the open hands ones first and only if your time-- uh, if you have time, go to the super emoji sections.

87. 17:25

    Uh, you'll also be filling code into a Jupyter Notebook, uh, and the places where you need to fill in code are indicated by these code_block placeholders. So those are the ones you need to fill in with your code.

88. 17:37

    And before any cell in the notebook that requires you to fill in code, you'll also see this books emoji indicating documentation that you need to reference for that particular piece of code.

89. 17:48

    And finally, you'll find, uh, solutions to all the hands-on pieces in the-- at the QR code link. But I highly encourage you to try working through stuff on your own before you look at the solutions.

90. 18:01

    And even if you do, then try to understand what's really going on.

91. 18:06

    With that, let's go ahead and break for our first hands-on section, uh, which is just setting up the development environment and prerequisites for the workshop. So yeah, let's take about fifteen to twenty minutes to work through this section.

92. 18:20

    Um, so if you go to that link, you want to start at the section titled MongoDB Atlas and work all the way through to the Dev Environment section. Let's go.

93. 18:33

    How are we feeling? Um, are we mostly done? Not done? Done? Like, show of hands, how many folks are done with this part?

94. 18:44

    Okay. Uh, mostly done? Okay. Uh, five more minutes? Yeah, let's do five more minutes.

95. 18:56

    All right. I think I'm gonna move on just in the interest of time, but it's a self-paced lab and you have access to all the material after the fact, so feel free to move at your own pace.

96. 19:07

    Um, cool. So let's move on to some libraries, tools, and general concepts that you'll come across in the next hands-on portion. So the first thing you'll run into is, um, this library called Datasets, which are go- we are going to use to download a dataset of arXiv papers from Hugging Face.

97. 19:25

    Uh, we are going to use the load_dataset method to download the arXiv embeddings dataset from, uh, the MongoDB educational AI Hugging Face org.

98. 19:36

    And then you'll run into something called arXivLoader, which is a document loader class in LangChain. Uh, we are going to be using this to load research papers from arXiv org as LangChain document objects.

99. 19:48

    And an example of what a document in LangChain looks like is shown here. So essentially has the raw text under the page_content attribute and some automatically extracted metadata, in this case, the publish date, title, authors, and summary under the metadata attribute.

100. 20:07

     So we're gonna be using arXivLoader in one of our-- two of our agent tools. One tool is already done for you, and that's the tool to get relevant papers from arXiv, and you'll also use the same, um, document loader to-- for the summary tool as well.

101. 20:22

     So the simplest way to create tools in LangChain is using the tool decorator, which makes tools out of functions. So for this tool, we have used the load method of arXivLoader to load data into document objects, and the query argument takes a topic or paper ID, and the load_max_docs indicates how many documents to download from arXiv.

102. 20:42

     And finally, we are only extracting the metadata because we want to only provide a list of papers, uh, and not the full paper content.

103. 20:51

     We'll also be using PyMongo, which is the Python driver for MongoDB. We'll use it to connect to MongoDB databases and collections, and also delete and insert documents, uh, from and to MongoDB to build the knowledge base for our agent.

104. 21:07

     Uh, we'll also be using a few LangChain integrations, which are essentially standalone packages for, uh, third-party providers such as MongoDB and, uh, LangChain to make things like versioning, dependency management, and testing kind of easier.

105. 21:20

     Uh, so we'll use the LangChain MongoDB integration to use MongoDB Atlas as a vector store, and also to store and retrieve chat history for the agent. We'll also use, uh, langchain-huggingface to access open source embedding models from Hugging Face.

106. 21:37

     And finally, we'll use langchain-fireworks to access, uh, chat completion models from Fireworks AI.

107. 21:45

     And you'll be using the LangChain expression language or LCEL to create RAG and agent workflows using LangChain, and it's essentially a declarative way to chain together prompts, data processing steps, LLMs, and tools in a LangChain fashion.

108. 22:02

     Uh, and each unit in the chain is called a runnable, and the way to chain them together is using the pipe operator that takes the output from the left of the pipe and passes it as input to the right of the pipe.

109. 22:14

     Uh, and here's a simple example of just passing a prompt to an LLM, generating an answer, and formatting its input. And finally, if you want to call the chain, then you use the inv-invoke method on it.

110. 22:25

     And you'll be using this to test out some of the things that you're building during the workshop.

111. 22:31

     And finally, you have this thing called RunnableLambda. Uh, and this is a runnable that converts any arbitrary Python function into a LangChain runnable, and it's as simple as defining the function and then wrapping the function into a RunnableLambda.

112. 22:47

     So yeah. Let's take another twenty minutes to now create the tools for your research agent. Uh, so yeah. Just work through the Create Agent Tools section of the lab that you were, uh, just working through.

113. 23:01

     Okay. Um, so hopeful-hopefully, we are kind of at least midway through creating tools for our agent. But in the next section, we are gonna be creating, uh, the agent itself and experiment with, um, the different reasoning design patterns that we were talking about, like chain-of-thought and ReAct.

114. 23:20

     So to create the agent, we are gonna start with the simplest way of creating a tool calling agent in LangChain, which is using the create tool calling agent constructor.

115. 23:30

     Um, and you're gonna be us- starting with that abstraction, but let's try to understand what's happening behind the scenes of that abstraction, right? So it's essentially creating a runnable sequence consisting of a prompt template, which has a placeholder for the agent scratchpad, which is the agent's intermediate steps as it's taking different actions and making observations, uh, an

116. 23:51

     LLM with knowledge of the tools that we were just creating, and an output parser for formatting the agent's response.

117. 24:00

     And then we'll also be exploring a ReAct agent that uses ReAct prompting to guide the agent to take a series of reasoning and action-taking steps to arrive at the final answer.

118. 24:11

     And for this, we'll use the create_react_agent constructor, which follows a similar series of steps as the tool calling agent, except it uses a ReAct prompt template, and the LLM has a knowledge of when to stop the reason action-taking sequence using, uh, a stop sequence.

119. 24:29

     And then the output parser has logic to parse these ReAct style LLM calls, and you can see what those look like right there. So it has, um, a thought, an action, an action input, and an observation.

120. 24:41

     So just parsing that to make it more readable to the user in the end.

121. 24:47

     And finally, uh, you'll come across the agent executor, which is the runtime for the agent. This is what actually calls the agent, executes the action that the agent is choosing, passes the action outputs back to the agent, and repeats any steps as the agent decides what to do next.

122. 25:03

     And that's the pseudocode what-- for what the agent executor is essentially doing. So as long as the agent thinks that it hasn't finished its task, which is the while loop there, the agent determines and runs a series of actions until it finally finishes the task.

123. 25:21

     So yeah, let's take another, another twenty minutes to complete the create agent section and any other things that you were working on previously.

124. 25:31

     Yeah.

125. 25:32

     One more time.

126. 25:33

     All right. Uh, we have one last thing to do with our research agent, which is to give it memory or add short-term memory to it. Um, and in this case, we are going to do that by giving it access to its chat message history.

127. 25:50

     So in LangChain, the way to do this is by wrapping the agent runnable that you created using the create tool calling agent or create ReAct agent, wrapping that runnable inside another runnable called runnable with message history, which is specifically designed to manage the memory of other runnables.

128. 26:08

     So essentially, this runnable can take a function that persists the chat message history for your agent to a database. We'll use MongoDB in this case, and by default, it organizes the chat history using a session ID that you pass in, uh, along with your input query or prompt.

129. 26:26

     So yeah, let's play around with that for the remainder of the time, and if you have any more questions or stuck at something, we can talk through that too for the rest of the time.

130. 26:39

     One last thing I would request, uh, once you're done with all your stuff is, um, yeah, if you want to connect, that's not the, uh, mandatory thing. Uh, nothing is mandatory.

131. 26:50

     But yeah, I'd really appreciate if you could fill out a short survey that's at the QR code link that you, uh, scanned in the beginning. This is the first time I'm doing this workshop, so any feedback you have will only help me make this, uh, better in the future.

132. 27:05

     So yeah, that'd be much appreciated. Um, other than that, this is it from me for today, and thanks for being here. [upbeat music]