Let's Build an Agent from Scratch — Kam Lasater

Talk transcript

1. 0:01

   Hi, I'm Kam, and I welcome you to my talk on how to build an agent. So I have three goals for this talk. I want you to experience the simplest version of what an agent could be.

2. 0:15

   Um, and I want you to feel comfortable in running and breaking the included code. Um, I think the, the running and breaking is a very, very important part to the learning.

3. 0:26

   Um, and I also want you to take away, uh, an intuition for how agents work or how other frameworks work. If you choose to go with some agent framework, that's great.

4. 0:38

   Um, I want you to have some understanding of maybe the underlying, the underpinning building blocks that they used to make it work. So with that, let's get to some slides.

5. 0:48

   Okay, great. This is where you can find me on LinkedIn. This is where you can find the slides for this talk and the code. I really encourage you to go and grab the code, try it, run it, break it, see where this code begins to turn from deterministic outcomes to a feeling that you're starting to, starting to

6. 1:09

   play with an agent. So what is an agent? Uh, I really like both of these definitions. Uh, agent, LLM, memory, planning, tools, and a while loop. Um, so let's break that down a little bit more.

7. 1:19

   So mathematically, we could, uh, take memory and say, "Oh, actually, this is a, both a read and a write operation," and also on the while loop, um, it's really a conditional and a looping.

8. 1:30

   So we can, uh, break those apart, reorder them a little bit, and come up with the plan for what we're gonna do. And so with all of that out of the way, let's jump in and get to the code finally.

9. 1:41

   Alrighty, so let's jump right in. So s- the first step here is, uh, calling an LLM. This is straight from the OpenAI, uh, Hello World docs, right? Uh, standard chat completion, and we can run it here.

10. 1:58

    Step zero. Okay. Um, this should be pretty standard for most people so far. Okay, great. Um, then let's jump to the step one, the condition. So, um, here we're gonna have the same completion call, um, checking for the prompt, but then once we get the answer back, we're gonna make another

11. 2:23

    LLM call that's gonna work as an LLM as judge, right? We are gonna ask, "You're a strict critic. Given the following question, determine if the answer is a full answer to the question."

12. 2:35

    Question, answer. Okay. Um, and then we give it a forced, um, JSON strict response of, um, whether it's done or not. And you can see the object here is whether it's done, it's gonna be a Boolean.

13. 2:50

    And the original question is, what is the average wing speed of a swallow? So we can go in, and we can run step one to see how it works with a condition.

14. 3:03

    So, um, it gives me LLM as judge, gives me a thumbs up. That's great. [clears throat] Here's the answer coming back from the LLM. So, so far, very deterministic, very mechanistic in its, uh, outputs, and then the judgment, and then the response to the, to the user.

15. 3:19

    So, um, the next step here is tools. Um, and we are incorporating SerpApi. I think I'm pronouncing that correctly. This is a Google Search API, uh, service. Um, and they basically make Google Search API, uh, easy JSON, um, API versus dealing with whatever the, the Google API nonsense is. [clears throat]

16. 3:43

    So I'm asking about buying a hoodie in New York and, um, near Times Square and the like. Again, gonna have the same conditional. I'm gonna now have this searchGoogle tool, right?

17. 3:55

    And you can see the Serp, um, getJSON call here. Engine is Google, passing in the API key with a query, um, and a location. I'm defaulting to Philadelphia from the location of where to start the query from. [clears throat]

18. 4:13

    And then I'm gonna print the results. Um, the tool config going into OpenAI looks like this, and so I'm, I'm hand writing out this JSON so that we can inspect it a little bit.

19. 4:25

    Um, we're defining the name of the function we're gonna call, giving it descriptions, um, strict, whether it, uh, is strictly adheres to the, the schema here, um, and then the parameters into the tool call.

20. 4:38

    So there's a query and a location, um, and which elements are required, and if there's any additional ones. So this is the JSON that gets generated and passed in, um, when we make our call.

21. 4:50

    Um, the other thing to note with tool calls

22. 4:54

    is that we call the LLM, the tool call kicks back and tells our local code what tool to call or which tools to call, and what, uh, parameters are included in there.

23. 5:06

    So we need to handle that. That's not handled by the OpenAI SDK, um, and we need to look into the tool call. If there are tool calls, in this case, we're only doing a single one.

24. 5:16

    We'll get to parallel, uh, in a few minutes. Um, and then we go, and we make that tool call, um, off of, uh, an array here on tools, which includes the searchGoogle, um, passing in the args.

25. 5:32

    And then we, uh, push those, both the tool call and the response from the tool call back to the conversation, and then we can, um, complete with tools again, meaning in this case, we are recursing through and calling, uh, back to the LLM with those responses passed in as those args, which it'll then make another determination.

26. 5:54

    So, um, so main loop here. Um, complete with tools, the prompt Search Google tool config, and then, um, it re-responds back, and then it'll see, it'll internally loop on tool calls until it is satisfied and the LLM is satisfied, and then result come back out.

27. 6:15

    And then here, the critic will come and perform, uh, condition again. So let us go here. Let us run step two on the tool.

28. 6:31

    Unhandled rejection. Uh-oh. Okay, it looks like we failed there due to, um, validation constraint on the location parameter.

29. 6:46

    It doesn't take just any string, it takes a specific set of strings. Um, in this case, the LLM-as-judge gave us a thumbs up again. Um, again, this feels very deterministic.

30. 6:57

    We are, um, asking an LLM to determine that it needs to call a search API, and then we're asking the same LLM to go and evaluate whether that search API returned text that was a reasonable answer to where I could buy a fur-lined hoodie near Times Square.

31. 7:16

    So, um, of course, still very mechanistic, very, um, deterministic, very, um, straightforward in its, in its outcomes. Um, okay, but then let's take the next step. Let's go to step three.

32. 7:30

    And in this case, um, a little refactoring is in order. Um, so same prompt around, um, wanting to buy a, a hoodie, fur-lined hoodie. Um, we've pushed the complete with tools into a other, into a utils folder, so into another file, um, just to get it out of that main.

33. 7:52

    This still has that same looping, but in this case, we've written this so that it can do the parallel tool calling. Um, these tool calls could come back with multiple, right?

34. 8:03

    This could be an array of tool calls that the LLM is asking the, the, us to perform, the client to perform. And so we're, uh, promising all over the tool calls, and we are, um, passing in the function arguments for each of the tool calls, um, into the tool functions, um, here.

35. 8:27

    And then we are then pushing the response from this local function that we have called back into, uh, the conversation. And this will have a tool call ID so that the LLM is able to trace, um, I asked for this tool to be called, and here's the ID, and then here's the response of that tool call.

36. 8:47

    So sometimes it asks for the same tool to be called multiple times with different input parameters, right? So that's how that works. Um, the tools, um, are configured. So for example, in the search Google example,

37. 9:02

    we're passing in this query, um, as an object, and it's performing the query, and it's returning a string result. Um, in my mental model, uh, the tools are, are best handled by thinking of them as text transformations.

38. 9:19

    You might have a couple of different parameters going in, but usually string to string is sort of the core of what, uh, these tools are performing. Um, so that was the AI and the, the completions, uh, the, sorry, the complete with tools on the refactor.

39. 9:38

    Um, and so that is here. And then we are outputting the result, um, and then we are running the same LLM as judge. So we can run that again.

40. 9:51

    That'll give us the step three. Um, that'll give us some additional looping and parallel tool calling.

41. 10:08

    All righty. LLM as judge, thumbs up again. Okay, great. Here are a couple of those sites. Again, though, still feels very deterministic. We just reshuffled things around. Shouldn't have been too big of a difference.

42. 10:19

    This is where we are gonna really feel an inflection here. So, um, the biggest change on step four for planning is creating a to-do list, and this now solves both the read, the write, and the planning aspect.

43. 10:36

    Um, the while looping aspect that we talked about is covered by the LLM itself, right? In making a tool call and then having us reply back with the result of that tool call, the LLM is able to keep iterating and keep, um, operating on the code that we are, or the prompt that we gave it, and keep

44. 11:01

    working through towards a solution. Um, there are cases when that LLM can just iterate forever, calling tools and never converging, as particularly if we keep pruning the context window and we never hit some error or boundary condition.

45. 11:18

    Um, some guardrails I've seen get put on are the number of, uh, iterative loops that they're able to go through before, um, [chuckles] client code kinda cuts them off and says like, "Okay, LLM, uh, you're drunk.

46. 11:31

    Go home." Um, so in this case though, we don't have that set up, um, but we have added this to-do list, which has what you would expect from a standard hello world to-do list, right?

47. 11:45

    You can add things to your to-do list, and in this case, we have the, also the tool config for the to-do list of adding new to-dos. That's an array of to-dos, right?

48. 11:54

    There'll be an array of to-dos, and we'll have array of done. Um, so you can add new to-dos, and they get pushed onto the to-do list. And then we're just printing it out, and then we're returning the to-dos that were added to the to-do list.

49. 12:08

    Um. Then we can also mark to-dos as done. Um, and so if a to-do is included in this to-do list,

50. 12:18

    um, mark it as done and pull it out of the, um, to-dos that are there, um, so that we don't have to do it again. Um, as well as the config for that.

51. 12:31

    Um, and then the check done. We can see if, uh, we have completed all of the to-dos. Um, if the length is zero, then it'll say, "No tasks have been marked done."

52. 12:45

    Um, and then the config for that as well, and then check to-dos. So it's able to get all the to-dos that are on the to-do list. So again, sort of standard to-do list.

53. 12:56

    Um, and now with all of those tools, the, the LLM-as-judge, the search Google, and I guess I should cover this, this browse web. Very similar to the search Google, right?

54. 13:09

    It takes in a URL. Um, it's using Cheerio and Turndown, and we are, uh, requesting that URL here. Um, and if the response comes back, uh, 200, then it's just taking the text

55. 13:24

    using Turndown to turn it into markdown and returning that markdown, um, out of the tool. So very simple operation given that URL, right? So given that, the planning will then take the to-do list that is generated based on the prompt, right?

56. 13:42

    So this is where the programming of the agent begins to take over, right? You're a helpful assistant working for a busy executive. Your tone is friendly but direct. They prefer short, clear, and direct writing.

57. 13:53

    You try to accomplish the specific task you're given. You can use any of the tools available to you. Before you do any of your work, you always make a plan to use your to-do list, right?

58. 14:02

    So that's driving the planning towards the to-do list. Uh, you can mark to-dos off of your to-do list after they've been complete. You summarize the actions you took by checking the to-do list, then create a report.

59. 14:13

    You always ask your assistant to check goal done. That drives towards LLM-as-judge. If you say you are done, you send the report to the user. If your assistant has feedback, you add it to your to-do list.

60. 14:24

    And then I've added today's date because sometimes the LLM doesn't quite know when in time it is, [chuckles] which is quite amusing. And in this case, since we've pushed LLM-as-judge as a tool, right, the condition can be a tool itself.

61. 14:38

    The loop is handled by the LLM itself. Then this main function is just a single call to this complete with the tools, and then we're gonna respond with our answer.

62. 14:48

    So, um, oh, and the, the default here is I want to learn about building agents without a framework. So let us try that. We can npm run step four and see how it goes.

63. 15:03

    I want to learn about building agents without a framework. Okay, it's calling the add to-dos. So here it came up with a plan. These are the new to-dos, right?

64. 15:12

    Um, search for information about building agents without a framework. Summarize key points from the search results, and then he asks the assistant to check if the expl-explanation is sufficient.

65. 15:21

    Um, it's calling check to-dos, calling search Google about this. It's browsing this blog post. It's browsing this Pond data, browsing the web, browsing the web. Here's a bunch of markdown from that website.

66. 15:39

    Um, it's marking done, summarizing key points from the search result. It's checking to-dos. It's checking goal done.

67. 15:50

    It's giving it, uh, LLM-as-judge gives it a thumbs up. Okay, great. Thumbs up. We're, we are done. Oh, so here is a summary of building an agent without using a framework.

68. 15:59

    To build an agent without a framework, follow these steps. Understand co-core components. Recognize that an AI agent is a language model capable of tool use and maintaining conversational context.

69. 16:08

    Define tools. Tools are functions for environmental interaction, um, like database queries, web search. Okay. That's tools. That's memory. Um, a loop, uh, input processing, tool decision, execution, response formulation, prompt engineering, direct API calls.

70. 16:27

    Uh, this is a good one instead of trying to pass through the LLM, but just directly store state and call it for yourself. Um, so I feel like this is pretty decent.

71. 16:35

    Um, you know, where I've started to see some, um, real interest in this LLM, uh, or this agent that I built is, uh, doing some things like, uh, I am planning a date with my wife this Saturday,

72. 16:57

    um, near [REDACTED:location]. Please help, help me find

73. 17:11

    some activities and dinner between 6 PM and 11 PM.

74. 17:23

    Um, let's see how it does. Um, so searching for local activities. Here's the plan again. Finding a good dinner option.

75. 17:35

    Um, searching Google. Best restaurants. Browsing the web.

76. 17:44

    Lots of markdown. Marking things as done. Checking the goal.

77. 17:54

    Okay. Here are some of the activities. Here are the, some of the dinner options, and hopes that I have a lovely evening. So it's not that this, uh, this agent is perfect by any means, but hopefully, you can see how adding each of these components from the LLM call, the conditional, the tool use,

78. 18:19

    um, adding some planning, some read/write in that to-do list, and how we can really leverage some of those things coming together and start to produce something that's, that's quite interesting.

79. 18:30

    You could see how next steps could be adding a vector database or injecting these, um, browsed, um, web pages into, um, you know, a, a Chroma DB in memory or into something, into a more RAG-like system.

80. 18:46

    So hopefully, this was really helpful, and, uh, hopefully, this gives you some interest or excitement in how to dive in and, uh,

81. 18:56

    do this yourself. Okay, great. Thanks for coming to my talk. Uh, again, I'm Kam Lasater. You can find me on LinkedIn here. The slides are up on my personal site, and the code is up on GitHub.

82. 19:09

    I encourage you to take it for a spin. Uh, let me know what you think. Let me know if, uh, there's some improvements you could make or if, uh, something like this is exciting.

83. 19:18

    We're always, uh, building and looking for people who are interested to build, uh, agents for, for our customers. So again, see you online. Cheers.