AI Engineer Summit 2025
Let's Build an Agent from Scratch — Kam Lasater
Read the talk
Build an Agent from a Completion, Tools, and a To-Do List
Kam Lasater builds an agent one capability at a time, showing how tool execution, completion checks, and mutable task state turn a simple model call into a planning workflow.
From a talk by Kam Lasater
Before you start: Familiarity with TypeScript, asynchronous functions, JSON, and a basic LLM API call will help you follow the implementation.
What has to surround a model call?
What is the smallest agent you can build well enough to run, break, and understand? That is the starting problem in Kam Lasater’s tutorial. The aim is to expose the machinery beneath an agent framework, so that using a framework becomes an informed choice. The companion code provides successive versions to experiment with as the behavior moves beyond a single request and response.
Start with an LLM, memory, planning, tools, and a while loop. Then split memory into read and write operations, and split the while loop into a condition and iteration. This decomposition gives the build its sequence: call the model, judge an answer, execute tools, support repeated calls, and finally add a plan that the agent can read and update. None of these components requires a framework to make its role visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Answer a question, then judge the answer
Step zero is an ordinary OpenAI Chat Completions call. Step one adds a second model call: a strict critic receives the original question and the proposed answer, then decides whether the answer fully addresses the question. Its response must follow a strict JSON schema containing a Boolean done field. This article retains the recording’s Chat Completions approach; current API documentation may use different request shapes.
The question is the average wing speed of a swallow. After the first model answers, the judge returns a thumbs-up. The application now has an answer and a machine-readable completion decision, but those are separate outputs with separate failure modes. A schema-valid Boolean is not proof that the answer is correct. Lasater describes this stage as deterministic and mechanistic: the workflow is fixed—generate, judge, respond—even though the model’s content is not guaranteed to be identical on each run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the model request a search
Step two changes the task to finding a fur-lined hoodie near Times Square in New York. To answer with outside information, the agent gets a searchGoogle tool backed by SerpApi. Its getJson call supplies the Google engine, an API key, a query, and a location. The implementation defaults the search location to Philadelphia, which is distinct from the location requested in the shopping task.
The tool has two interfaces. The local function performs the search and prints results. A handwritten tool configuration tells the model the function’s name, description, strictness, and parameters. It describes query and location, identifies required fields, and specifies whether additional properties are permitted. That schema is how the model learns what it may request; it does not execute the search.
The application executes the tool; the model requests it. The initial implementation handles one requested call at a time:
- Call the model with the conversation and tool configuration.
- Inspect the returned tool call for its function name and arguments.
- Dispatch the request to the local
searchGooglefunction through the tools collection. - Append both the assistant’s tool-call message and the tool’s response to the conversation.
- Call
completeWithToolsagain with the expanded conversation.
The OpenAI SDK does not automatically run these local functions. The recursive call gives the model the search result and another opportunity to choose what happens next.
Once the model stops requesting tools and produces an answer, completeWithTools returns it to the main flow. The external critic then checks that answer, just as it did for the swallow question. There are now two distinct decisions: whether another tool call is needed, and whether the resulting answer satisfies the user’s request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A valid-looking argument can still fail
The live search run produces an unhandled rejection. Lasater attributes it to a validation constraint on location: the value must come from a particular set of accepted strings, rather than being any arbitrary string. A parameter can therefore look reasonable to the model and still fail the service’s requirements.
Lasater also reports another judge thumbs-up, without explaining how that approval relates to the failed request. The approval cannot establish that the search succeeded. At this stage, the same LLM chooses to search and evaluates whether the returned text reasonably answers the hoodie question; the surrounding workflow still feels straightforward and mechanistic to him.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Execute multiple calls without losing their identities
Step three keeps the hoodie request but moves completeWithTools into a utility file. The helper now accepts an array of requested tool calls and executes them with Promise.all, passing each call’s arguments to the corresponding local function. Each result carries a tool call ID so the model can associate it with the request that produced it.
Function names alone are insufficient: the model may request searchGoogle several times with different parameters. In TypeScript, the dispatch portion can keep each request’s identity attached to its result like this:
typescript
type ToolCall = {
id: string;
function: { name: string; arguments: string };
};
type Tool = (args: Record<string, unknown>) => Promise<string>;
type ToolResult = {
role: "tool";
tool_call_id: string;
content: string;
};
async function executeToolCalls(
calls: ToolCall[],
tools: Record<string, Tool>,
): Promise<ToolResult[]> {
return Promise.all(
calls.map(async (call): Promise<ToolResult> => {
const tool = tools[call.function.name];
if (!tool) {
throw new Error(`Unknown tool: ${call.function.name}`);
}
const args = JSON.parse(call.function.arguments);
const content = await tool(args);
return {
role: "tool",
tool_call_id: call.id,
content,
};
}),
);
}
The surrounding helper appends these results after the assistant message that requested the calls, then asks the model to continue. The ID preserves the relationship even when the same function appears more than once.
Lasater’s preferred mental model for a tool is a text transformation. searchGoogle accepts a query object and returns a string; even when inputs have several fields, the model-facing interaction is largely text in and text out. The refactored run returns sites and another judge approval. Parallel dispatch improves the machinery, but it has not yet produced the behavioral change he is looking for.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A to-do list supplies planning and memory
Step four adds a to-do list, the point Lasater identifies as the behavioral inflection. The same mechanism now supports planning, reading state, and writing state. Iteration still comes from the exchange between model and application: the model requests another tool, application code executes it and sends back the result, and a subsequent model call decides how to proceed.
That exchange does not guarantee convergence. A model can keep requesting tools indefinitely, especially if context pruning prevents it from reaching a context boundary or error. Lasater describes client-enforced iteration limits as a possible guardrail, but the demonstration does not implement one. Moving the next-action decision into the model does not remove the application’s responsibility for bounding execution.
The storage is deliberately ordinary: one array holds pending to-dos, and another holds completed items. The add operation accepts an array of tasks, pushes them into pending work, prints them, and returns the tasks that were added. It also has a tool configuration so the model can request the operation.
The remaining operations make that state usable across successive model calls:
| Operation | Effect |
|---|---|
| Mark a to-do done | Remove an existing task from pending work and record it as done |
| Check done | Inspect completed work; report when no tasks have been marked done |
| Check to-dos | Retrieve the pending list |
Each operation has a corresponding tool configuration. Removing completed tasks from the pending list helps prevent repeated work. Inspecting the completed list is also distinct from deciding whether the user’s overall goal has been satisfied.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the tools a working procedure
The available tools now include the judge, search, the to-do operations, and browseWeb. The browsing tool takes a URL, retrieves its content, and uses Cheerio and Turndown to return Markdown after an HTTP 200 response. This is page retrieval and conversion, not a full browser that renders a page and executes its JavaScript. The result is text the model can use in its next decision.
The prompt then supplies a procedure for using those capabilities. It casts the model as an assistant to a busy executive, with friendly, short, clear, direct writing. More consequentially, it tells the model how to organize the work:
- Make a plan in the to-do list before doing any work.
- Use available tools to carry out the task.
- Mark tasks complete after doing them.
- Check the list and create a report of the actions taken.
- Ask the assistant to check whether the goal is done.
- Send the report if approved; otherwise, add the feedback to the to-do list.
The list stores the work; the prompt establishes when and why to update it.
Lasater also includes the current date to orient the model in time. The judge has moved from a fixed call after the main workflow into the available tool set, so the completion condition itself becomes something the model is instructed to invoke. The main function can now make one call to completeWithTools and return its final answer. That compact entry point still wraps multiple model requests and local tool executions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Watch the agent research how to build agents
The default step-four request is to learn about building agents without a framework. Lasater runs the step through npm, and the model first calls the add-to-dos tool. Its generated plan has three parts: search for information, summarize the key points, and ask the assistant whether the explanation is sufficient. The model is now choosing intermediate work and recording it before producing the final answer.
The run checks the to-do list, calls searchGoogle, and browses multiple pages. Markdown comes back into the conversation. The agent then marks work done, checks its to-dos, and requests the goal check. The judge’s thumbs-up precedes the final report. This sequence makes the planning cycle visible through ordinary tool calls and state changes.
The resulting report describes an agent as a language model with tool use and conversational context. It identifies tools such as database queries and web search, then outlines input processing, tool decisions, execution, response formulation, prompt engineering, and direct API calls. Lasater particularly endorses the suggestion to store state directly and access it in code instead of passing everything through the LLM. The to-do arrays already embody that choice: the model requests changes, while local code owns the stored state.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try a broader task, then extend the same machinery
For a broader test, Lasater asks for help planning a Saturday date with his wife near RVA, with activities and dinner between 6 PM and 11 PM. The generated plan separates finding local activities from finding a dinner option. It searches Google for restaurants, browses pages, receives Markdown, marks work done, and checks the goal.
The final response offers activity and dinner options. Lasater does not present the agent as perfect; what interests him is how the components now work together. A model call and a condition established the initial workflow. Tools connected it to outside information. Planning and the to-do list’s read/write operations gave it a way to organize and track intermediate work on a less narrowly specified request.
A possible next layer would retain the browsed pages in a vector database—perhaps an in-memory Chroma database—or use them in a retrieval-augmented system. These are proposed extensions, not capabilities demonstrated in the run. They would build on an existing path from URL to page text, adding a way to retrieve that information later.
The exercise ends with an invitation to run the code, break it, and suggest improvements. Lasater points readers to the slides on his personal site, the code on GitHub, and LinkedIn for contact. Having watched each layer appear, the useful experiment is to change one of them and observe how the agent’s behavior changes: the implementation is small enough that the relationship between a tool, a prompt instruction, and the resulting work remains visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The February 2025 tutorial repository, progressing from an LLM call to tools, to-do planning, memory, and iteration.
Google search results through a JSON interface, with query and geographic-location parameters.
Parse and extract HTML using a jQuery-like interface without running a browser.
A JavaScript library for converting HTML strings and DOM nodes into Markdown.
Updates since the talk
Current documentation for defining functions, executing model-requested calls, and returning results to the model.
Current guidance for schema-constrained responses, including their remaining correctness limitations.
Read the complete timestamped transcript
- 0:01
Hi, I'm Kam, and I welcome you to my talk on how to build an agent. So I have three goals for this talk. I want you to experience the simplest version of what an agent could be.
- 0:15
Um, and I want you to feel comfortable in running and breaking the included code. Um, I think the, the running and breaking is a very, very important part to the learning.
- 0:26
Um, and I also want you to take away, uh, an intuition for how agents work or how other frameworks work. If you choose to go with some agent framework, that's great.
- 0:38
Um, I want you to have some understanding of maybe the underlying, the underpinning building blocks that they used to make it work. So with that, let's get to some slides.
- 0:48
Okay, great. This is where you can find me on LinkedIn. This is where you can find the slides for this talk and the code. I really encourage you to go and grab the code, try it, run it, break it, see where this code begins to turn from deterministic outcomes to a feeling that you're starting to, starting to
- 1:09
play with an agent. So what is an agent? Uh, I really like both of these definitions. Uh, agent, LLM, memory, planning, tools, and a while loop. Um, so let's break that down a little bit more.
- 1:19
So mathematically, we could, uh, take memory and say, "Oh, actually, this is a, both a read and a write operation," and also on the while loop, um, it's really a conditional and a looping.
- 1:30
So we can, uh, break those apart, reorder them a little bit, and come up with the plan for what we're gonna do. And so with all of that out of the way, let's jump in and get to the code finally.
- 1:41
Alrighty, so let's jump right in. So s- the first step here is, uh, calling an LLM. This is straight from the OpenAI, uh, Hello World docs, right? Uh, standard chat completion, and we can run it here.
- 1:58
Step zero. Okay. Um, this should be pretty standard for most people so far. Okay, great. Um, then let's jump to the step one, the condition. So, um, here we're gonna have the same completion call, um, checking for the prompt, but then once we get the answer back, we're gonna make another
- 2:23
LLM call that's gonna work as an LLM as judge, right? We are gonna ask, "You're a strict critic. Given the following question, determine if the answer is a full answer to the question."
- 2:35
Question, answer. Okay. Um, and then we give it a forced, um, JSON strict response of, um, whether it's done or not. And you can see the object here is whether it's done, it's gonna be a Boolean.
- 2:50
And the original question is, what is the average wing speed of a swallow? So we can go in, and we can run step one to see how it works with a condition.
- 3:03
So, um, it gives me LLM as judge, gives me a thumbs up. That's great. [clears throat] Here's the answer coming back from the LLM. So, so far, very deterministic, very mechanistic in its, uh, outputs, and then the judgment, and then the response to the, to the user.
- 3:19
So, um, the next step here is tools. Um, and we are incorporating SerpApi. I think I'm pronouncing that correctly. This is a Google Search API, uh, service. Um, and they basically make Google Search API, uh, easy JSON, um, API versus dealing with whatever the, the Google API nonsense is. [clears throat]
- 3:43
So I'm asking about buying a hoodie in New York and, um, near Times Square and the like. Again, gonna have the same conditional. I'm gonna now have this searchGoogle tool, right?
- 3:55
And you can see the Serp, um, getJSON call here. Engine is Google, passing in the API key with a query, um, and a location. I'm defaulting to Philadelphia from the location of where to start the query from. [clears throat]
- 4:13
And then I'm gonna print the results. Um, the tool config going into OpenAI looks like this, and so I'm, I'm hand writing out this JSON so that we can inspect it a little bit.
- 4:25
Um, we're defining the name of the function we're gonna call, giving it descriptions, um, strict, whether it, uh, is strictly adheres to the, the schema here, um, and then the parameters into the tool call.
- 4:38
So there's a query and a location, um, and which elements are required, and if there's any additional ones. So this is the JSON that gets generated and passed in, um, when we make our call.
- 4:50
Um, the other thing to note with tool calls
- 4:54
is that we call the LLM, the tool call kicks back and tells our local code what tool to call or which tools to call, and what, uh, parameters are included in there.
- 5:06
So we need to handle that. That's not handled by the OpenAI SDK, um, and we need to look into the tool call. If there are tool calls, in this case, we're only doing a single one.
- 5:16
We'll get to parallel, uh, in a few minutes. Um, and then we go, and we make that tool call, um, off of, uh, an array here on tools, which includes the searchGoogle, um, passing in the args.
- 5:32
And then we, uh, push those, both the tool call and the response from the tool call back to the conversation, and then we can, um, complete with tools again, meaning in this case, we are recursing through and calling, uh, back to the LLM with those responses passed in as those args, which it'll then make another determination.
- 5:54
So, um, so main loop here. Um, complete with tools, the prompt Search Google tool config, and then, um, it re-responds back, and then it'll see, it'll internally loop on tool calls until it is satisfied and the LLM is satisfied, and then result come back out.
- 6:15
And then here, the critic will come and perform, uh, condition again. So let us go here. Let us run step two on the tool.
- 6:31
Unhandled rejection. Uh-oh. Okay, it looks like we failed there due to, um, validation constraint on the location parameter.
- 6:46
It doesn't take just any string, it takes a specific set of strings. Um, in this case, the LLM-as-judge gave us a thumbs up again. Um, again, this feels very deterministic.
- 6:57
We are, um, asking an LLM to determine that it needs to call a search API, and then we're asking the same LLM to go and evaluate whether that search API returned text that was a reasonable answer to where I could buy a fur-lined hoodie near Times Square.
- 7:16
So, um, of course, still very mechanistic, very, um, deterministic, very, um, straightforward in its, in its outcomes. Um, okay, but then let's take the next step. Let's go to step three.
- 7:30
And in this case, um, a little refactoring is in order. Um, so same prompt around, um, wanting to buy a, a hoodie, fur-lined hoodie. Um, we've pushed the complete with tools into a other, into a utils folder, so into another file, um, just to get it out of that main.
- 7:52
This still has that same looping, but in this case, we've written this so that it can do the parallel tool calling. Um, these tool calls could come back with multiple, right?
- 8:03
This could be an array of tool calls that the LLM is asking the, the, us to perform, the client to perform. And so we're, uh, promising all over the tool calls, and we are, um, passing in the function arguments for each of the tool calls, um, into the tool functions, um, here.
- 8:27
And then we are then pushing the response from this local function that we have called back into, uh, the conversation. And this will have a tool call ID so that the LLM is able to trace, um, I asked for this tool to be called, and here's the ID, and then here's the response of that tool call.
- 8:47
So sometimes it asks for the same tool to be called multiple times with different input parameters, right? So that's how that works. Um, the tools, um, are configured. So for example, in the search Google example,
- 9:02
we're passing in this query, um, as an object, and it's performing the query, and it's returning a string result. Um, in my mental model, uh, the tools are, are best handled by thinking of them as text transformations.
- 9:19
You might have a couple of different parameters going in, but usually string to string is sort of the core of what, uh, these tools are performing. Um, so that was the AI and the, the completions, uh, the, sorry, the complete with tools on the refactor.
- 9:38
Um, and so that is here. And then we are outputting the result, um, and then we are running the same LLM as judge. So we can run that again.
- 9:51
That'll give us the step three. Um, that'll give us some additional looping and parallel tool calling.
- 10:08
All righty. LLM as judge, thumbs up again. Okay, great. Here are a couple of those sites. Again, though, still feels very deterministic. We just reshuffled things around. Shouldn't have been too big of a difference.
- 10:19
This is where we are gonna really feel an inflection here. So, um, the biggest change on step four for planning is creating a to-do list, and this now solves both the read, the write, and the planning aspect.
- 10:36
Um, the while looping aspect that we talked about is covered by the LLM itself, right? In making a tool call and then having us reply back with the result of that tool call, the LLM is able to keep iterating and keep, um, operating on the code that we are, or the prompt that we gave it, and keep
- 11:01
working through towards a solution. Um, there are cases when that LLM can just iterate forever, calling tools and never converging, as particularly if we keep pruning the context window and we never hit some error or boundary condition.
- 11:18
Um, some guardrails I've seen get put on are the number of, uh, iterative loops that they're able to go through before, um, [chuckles] client code kinda cuts them off and says like, "Okay, LLM, uh, you're drunk.
- 11:31
Go home." Um, so in this case though, we don't have that set up, um, but we have added this to-do list, which has what you would expect from a standard hello world to-do list, right?
- 11:45
You can add things to your to-do list, and in this case, we have the, also the tool config for the to-do list of adding new to-dos. That's an array of to-dos, right?
- 11:54
There'll be an array of to-dos, and we'll have array of done. Um, so you can add new to-dos, and they get pushed onto the to-do list. And then we're just printing it out, and then we're returning the to-dos that were added to the to-do list.
- 12:08
Um. Then we can also mark to-dos as done. Um, and so if a to-do is included in this to-do list,
- 12:18
um, mark it as done and pull it out of the, um, to-dos that are there, um, so that we don't have to do it again. Um, as well as the config for that.
- 12:31
Um, and then the check done. We can see if, uh, we have completed all of the to-dos. Um, if the length is zero, then it'll say, "No tasks have been marked done."
- 12:45
Um, and then the config for that as well, and then check to-dos. So it's able to get all the to-dos that are on the to-do list. So again, sort of standard to-do list.
- 12:56
Um, and now with all of those tools, the, the LLM-as-judge, the search Google, and I guess I should cover this, this browse web. Very similar to the search Google, right?
- 13:09
It takes in a URL. Um, it's using Cheerio and Turndown, and we are, uh, requesting that URL here. Um, and if the response comes back, uh, 200, then it's just taking the text
- 13:24
using Turndown to turn it into markdown and returning that markdown, um, out of the tool. So very simple operation given that URL, right? So given that, the planning will then take the to-do list that is generated based on the prompt, right?
- 13:42
So this is where the programming of the agent begins to take over, right? You're a helpful assistant working for a busy executive. Your tone is friendly but direct. They prefer short, clear, and direct writing.
- 13:53
You try to accomplish the specific task you're given. You can use any of the tools available to you. Before you do any of your work, you always make a plan to use your to-do list, right?
- 14:02
So that's driving the planning towards the to-do list. Uh, you can mark to-dos off of your to-do list after they've been complete. You summarize the actions you took by checking the to-do list, then create a report.
- 14:13
You always ask your assistant to check goal done. That drives towards LLM-as-judge. If you say you are done, you send the report to the user. If your assistant has feedback, you add it to your to-do list.
- 14:24
And then I've added today's date because sometimes the LLM doesn't quite know when in time it is, [chuckles] which is quite amusing. And in this case, since we've pushed LLM-as-judge as a tool, right, the condition can be a tool itself.
- 14:38
The loop is handled by the LLM itself. Then this main function is just a single call to this complete with the tools, and then we're gonna respond with our answer.
- 14:48
So, um, oh, and the, the default here is I want to learn about building agents without a framework. So let us try that. We can npm run step four and see how it goes.
- 15:03
I want to learn about building agents without a framework. Okay, it's calling the add to-dos. So here it came up with a plan. These are the new to-dos, right?
- 15:12
Um, search for information about building agents without a framework. Summarize key points from the search results, and then he asks the assistant to check if the expl-explanation is sufficient.
- 15:21
Um, it's calling check to-dos, calling search Google about this. It's browsing this blog post. It's browsing this Pond data, browsing the web, browsing the web. Here's a bunch of markdown from that website.
- 15:39
Um, it's marking done, summarizing key points from the search result. It's checking to-dos. It's checking goal done.
- 15:50
It's giving it, uh, LLM-as-judge gives it a thumbs up. Okay, great. Thumbs up. We're, we are done. Oh, so here is a summary of building an agent without using a framework.
- 15:59
To build an agent without a framework, follow these steps. Understand co-core components. Recognize that an AI agent is a language model capable of tool use and maintaining conversational context.
- 16:08
Define tools. Tools are functions for environmental interaction, um, like database queries, web search. Okay. That's tools. That's memory. Um, a loop, uh, input processing, tool decision, execution, response formulation, prompt engineering, direct API calls.
- 16:27
Uh, this is a good one instead of trying to pass through the LLM, but just directly store state and call it for yourself. Um, so I feel like this is pretty decent.
- 16:35
Um, you know, where I've started to see some, um, real interest in this LLM, uh, or this agent that I built is, uh, doing some things like, uh, I am planning a date with my wife this Saturday,
- 16:57
um, near [REDACTED:location]. Please help, help me find
- 17:11
some activities and dinner between 6 PM and 11 PM.
- 17:23
Um, let's see how it does. Um, so searching for local activities. Here's the plan again. Finding a good dinner option.
- 17:35
Um, searching Google. Best restaurants. Browsing the web.
- 17:44
Lots of markdown. Marking things as done. Checking the goal.
- 17:54
Okay. Here are some of the activities. Here are the, some of the dinner options, and hopes that I have a lovely evening. So it's not that this, uh, this agent is perfect by any means, but hopefully, you can see how adding each of these components from the LLM call, the conditional, the tool use,
- 18:19
um, adding some planning, some read/write in that to-do list, and how we can really leverage some of those things coming together and start to produce something that's, that's quite interesting.
- 18:30
You could see how next steps could be adding a vector database or injecting these, um, browsed, um, web pages into, um, you know, a, a Chroma DB in memory or into something, into a more RAG-like system.
- 18:46
So hopefully, this was really helpful, and, uh, hopefully, this gives you some interest or excitement in how to dive in and, uh,
- 18:56
do this yourself. Okay, great. Thanks for coming to my talk. Uh, again, I'm Kam Lasater. You can find me on LinkedIn here. The slides are up on my personal site, and the code is up on GitHub.
- 19:09
I encourage you to take it for a spin. Uh, let me know what you think. Let me know if, uh, there's some improvements you could make or if, uh, something like this is exciting.
- 19:18
We're always, uh, building and looking for people who are interested to build, uh, agents for, for our customers. So again, see you online. Cheers.