Building Multimodal AI Agents (From Scratch)

[upbeat music] Hi, everyone.

Um, thanks for taking the time to be here today. Um, welcome to this workshop where you will learn about agents, multimodality, and hopefully get to build... Not hopefully, you will build, uh, a multimodal agent of your own from scratch.

Uh, whose first workshop of the day? How many? Show of hands. Okay, first workshop. How many people already been in a workshop? Oh, wow. Like really proactive, overachieving crowd.

Great to see that. All right. Um, with that, here's a little bit about me. I'm Apoorva. Uh, I'll be your lead instructor for today. I'm also joined by, um, my awesome team here.

We have Richmond. He's waving at you. There's Rafa back there. He's waving at you as well. There's Thibaut, and we have Makiko. Um, but yeah, g- here's a bit about me.

Uh, I'm Apoorva. I'm currently an AI-focused developer advocate at MongoDB, which means I spend a lot of my time building, uh, workshops like this one for AI developers like you, uh, to help them build AI applications of their own.

Uh, prior to this role, I spent about six years working as, uh, a data scientist in the cybersecurity space. And outside of work, I like to read a lot.

I try to yoga pretty regularly, or used to try to till I, uh, [laughs] busted my knee a while ago. Uh, and I'm always on a mission to, um, visit as many local coffee shops as I can in whichever city I'm in.

So this is what the next hour and twenty minutes is going to look like. Uh, we'll be going over key concepts of AI agents, discuss what multimodality is, and finally, we are gonna put the two together and build a multimodal AI agent from scratch using good old Python.

And since this is a relatively short workshop session, um, this is what you can expect. Uh, we are going to focus on getting concepts down, um, and we'll see what we have time for.

Based on my practice sessions, uh, I believe you get about fifty-five minutes to an hour to actually take your time to write code or just, like,

get a really good understanding of how, uh, multimodal agents work in code. So just wanted to, uh, set that expectation there.

And with that, let's get started. So let's first answer two questions. What are AI agents, and why do we need them? But even before I get into that, uh, how many of you already have some experience with AI agents?

Okay, there's some of you. Hopefully, there'll be something new for you, uh, at this workshop. And for those of you who are new to the concept, there's a lot of stuff coming your way.

So, uh, apologies for that, or not. Okay, so here's my definition of an AI agent. Um, I like to define an AI agent as a system that uses a large language model, I'll be referring to this as an LLM, uh, to reason through a problem, create a plan to solve, uh, the problem, and execute and iterate on

the plan with the help of a set of tools.

So in the past two years or so, uh, we have seen three main paradigms for interacting with LLMs. There's simple prompting, RAG, and agents. So let's briefly, uh, talk about each of these because this will help us build an intuition for, uh, when you want to use an AI agent versus something else.

So with simple prompting, you're simply asking, uh, the LLM questions and expecting the LLM to rely on its pre-trained, or as we call it, parametric knowledge, uh, to answer these questions.

So this means the LLM cannot answer questions, um, if the information required to answer them is not present in its pre-trained knowledge. It cannot really handle complex queries, and it cannot provide personalized responses or even refine its responses.

With RAG, uh, how many of you have built a RAG application? Okay, some of you. So, you know, with RAG, you take this up a notch. You augment the LLM's knowledge with information from external data sources.

And as you can imagine, this solves some problems where you can now be, uh, reasonably certain that the LLM has, uh, the information required to, uh, answer user questions and also incorporate some basic light personalization if it's given access to the right sources of information.

But this still doesn't equip the LLM with the ability to handle, uh, complex, uh, multi-step tasks or to self-refine its responses, but that's okay because not all tasks might require this capability.

And finally, uh, 2025 is the year of agents. So if you need-- uh, if you have complex multi-step tasks or need, uh, deep personalization, any sort of adaptive learning in your applications, then you'd want to use AI agents.

So with AI agents, what we've done is given LLM the agency to determine the sequence of steps required to, uh, complete a particular task. And they do this by taking actions with the help of tools that you provide them and reasoning through the result-results of these tool executions and also its past interactions, um, to inform what to

do next. And this is what makes agents extremely flexible and capable of handling a wide variety of complex tasks. But the... Yeah. Sorry, will you share these, uh, the presentation?

Yes, it will be shared. The you- uh, recordings will be up on YouTube as well. So yeah. Uh, the one thing to note here, though, is that agents come with, uh, higher cost and latency.

After all, you're expecting LLMs to do all the heavy lifting of, uh, thinking through the problem, coming up with a plan of action, uh, executing, um, the actions, rectifying its responses.

So my word of caution here is only use agents if you need to. Don't complicate whatever it is you're trying to build. But we are building an AI agent today, so for today, let's just throw an agent at the problem, a simple problem.

Okay. So to summarize, use agents for complex tasks that don't have a structured workflow, or where the series of steps required to solve the problem is hard to predict, or tasks that have a high latency tolerance, as I was just mentioning, or for tasks where it's acceptable for, uh, your application or system to return non-deterministic outputs, which

means, um, the same, uh, result is not guaranteed for the same out- inputs. And this is true for any application that uses LLMs, but this effect is especially amplified, uh, in agentic workflows.

And finally, tasks that might benefit from any sort of personalization or adaptive behavior over a long period of time, all of these are fair game for AI agents.

Now let's talk about the different components of AI agents, just to get a better understanding of, uh, how these systems work.

So an agent typically has four main components. There's perception, which is how, uh, agents gather information about their environment. Planning and reasoning, which helps, um, the agent reason through a problem and come up with a plan to solve it.

Then there's tools, which are external interfaces that help the agent act upon and solve a problem.

And memory, which helps agents learn from past interactions. And if you are passionate about memory, we have Rich Wind, who knows a lot about this topic, so definitely, uh, catch him, um, after or during the presentation.

So all of this sounds a bit like a human, doesn't it? But that's the whole goal of agents. The goal with, uh, LLM-based agents is to give, uh, these systems the autonomy to carry out complex tasks, much like we humans do.

So it's not a surprise that the components kind of resemble how we, um, think through problems and go about the world.

So let's dive a bit deeper into each of these components.

Let's talk about perception first. So perception, as I said, is the mechanism by which agents gather information about their environment. And this happens via some form of inputs, whether it's, uh, a user like you interacting with the agent or triggered by something else, like an email or a, a Slack message.

And text inputs have been the most common form of, uh, interacting with LLMs and agents so far. But over the past few months, we've seen, um, images, voice, video also being part of this, uh, perception mechanism for agents.

And in today's workshop, we'll be working with two of these, which is text and images.

The next component we have is planning and reasoning.

And shocker, [chuckles] the component that helps, uh, agents plan and reason is LLMs. So given a user query, it's the LLM's job to determine how to go about, uh, solving the problem.

But they can't do all of this on their own. They need some guidance, and the way to provide guidance, uh, at this point is to, um, prompt the LLM.

And you can start simple by prompting the LLM to create a plan of action based on its initial understanding of the problem, and this is what we call planning without feedback, since the LLM doesn't really modify, uh, its initial plan of action based on information gathered from tool outcomes or its own reasoning traces.

And a common design pattern for this kind of planning is chain of thought.

And chain of thought is as simple as prompting the LLM to think through a problem step by step without, um, directly jumping to, uh, giving the user an answer.

And you can do this in two ways, uh, in a zero-shot manner, where you literally prompt the LLM, like, tell it, like, "Let's think step by step," or you can do this in a few shot manner, where you're providing examples of how the LLM should go about thinking through a problem, so that the next time you give

it another problem, it'll, uh, use your examples to guide its reasoning process.

Then there's planning with feedback, where you can prompt the LLM to, uh, adjust and refine its, uh, initial plan based on new information, again, obtained from, uh, tool outcomes or based on its own, uh, previous reasoning.

And a common design pattern that you will implement today is, uh, ReAct, which is short for reasoning and act. And what we do in this pattern is you prompt the LLM to generate

verbal reasoning, uh, traces and also, uh, tell you the actions that it'll take to solve the task. And then after each action, we ask the LLM to make an observation based on, uh, the information that it gathered from that tool execution and think through that and plan what to do next.

And this continues until the LLM determines that, "I have the final answer," and that's when it'll exit that execution loop and provide the answer to the user.

Then the next thing we have is tools. And tools are essentially interfaces for, uh, agents to interact with their external world in order to, uh, achieve their objectives. And these tools can range from simple APIs such as, I'm sure you've seen, uh, examples of weather and search APIs, to, uh, vector stores, to even specialized machine learning models.

And tools for LLMs are typically defined as functions. And most recent LLMs have been trained to identify... Uh, when a function should be called, and also the arguments for a function call.

But the one thing to note is, um, the LLM doesn't actually execute the function. This is something, uh, we will have to implement in our code.

And in addition to actually defining the function, uh, you typically also need to provide, uh, the LLM a function or tool schema. And this basically is just a JSON file, or with, um, MCP servers, you might have seen a different way of, uh, defining, uh, tools.

But essentially, you're providing, um, the name of the tool to call, a description of what the tool does, and the pa-parameters that the tool takes, and also their types and descriptions.

So, for example, I have a weather tool here, and in my tool schema, I'm saying, uh, the name of the tool, the description, I'm saying, uh, the input that it takes as a city, the type is string, and, uh, the description of that parameter.

And finally, uh, the last component is memory.

This component is what allows AI agents to store and recall past conversations, uh, and enables them to learn from these interactions. And memory, if you think of human memory, it's a pretty nebulous concept.

There's so many different types of memory. If you ask a psychologist, they'll tell you, uh, all about that. But, uh, I'm not a psychologist, so [chuckles] uh, I think of it in pretty primitive terms.

Uh, I think of it in as two broad categories. Uh, one is short-term, which deals with storing and retrieving information from a single conversation, and then there's long-term, which deals with storing, updating, and retrieving information, um, ha- obtained over multiple conversations had with the agent over a longer period of time.

And this is really what, um, enables agents to personalize their responses over a long period of time. But in today's lab, we'll implement short-term memory for our multimodal agent.

And again, if you wanna learn more about this, uh, nebulous and extensive topic, then here's a talk that I gave a few months ago. Uh, so I'll leave this here for a few seconds, but if you wanna talk to someone live, then talk to me, Richmond, Mikiko, uh, anyone from our team.

Moving on. Okay. So let's take an example and understand how, how all of these components work together, right? So the first thing that happens is a query comes into the agent here.

Uh, I'm asking this agent what's the weather in San Francisco today?

The agent forwards the query to an LLM. So think of agent, uh, agents as a software application or system with different components, one of them being, uh, one or more LLMs.

And the LLM, in this case, has access to a set of tools. In this case, it has access to a weather and search API and also its past interactions or memories.

So based on the tools it has access to, in this case, for this query, the LLM might, uh, decide that the weather API would be the most suitable to get information about this query, and it'll also parse out the arguments for this tool from the user query.

And like I mentioned, the-- your agent also needs to have code to actually execute the tools, so we have that in our agent. It's going to make a call to the weather API, uh, with the arguments extracted by the LLM, get a response back from the API, forward that to the LLM.

And at this point, the LLM has two options. It can either decide that it needs to-- needs more information and decide to call more tools, or it can be like, "I have the final answer.

Uh, I'm gonna generate that now." So in this case, it has the temperature in San Francisco, so it might be like, "I know the answer." It generates a natural language response, and that gets forwarded to, uh, the user.

So that's kind of the full flow of how, uh, a tool calling, simple tool calling agent works.

The other thing we need to talk about today, which is the more interesting part, uh, I believe, is multimodality b-because we are, after all, building a multimodal agent.

So what is multimodality? Multimodality in the context of machine learning or AI is the ability of machine learning models to process, understand, and at this point, uh, even generate different types of, uh, data, such as text, images, audio, video, et cetera.

And like I mentioned, in today's lab, we'll be working with two of these modalities, which is text and images.

So here are some real-world examples of data that contains a combination of, uh, images and text, just to give you some inspiration for, uh, the kind of problems and domains you can apply your learnings from today too.

So there's graphs, tables, and then there's these types of data interweaved with text. So think of research papers, financial reports, any sort of organizational reporting, uh, which typically has, like, some graphs, analysis, and text, um, all combined together or healthcare documents.

The list is virtually endless. There's a lot of, uh, a lot of real-world, um, data looks something like this.

So to make sense of this type of data, we currently have two classes of multimodal machine learning models. And the first type of models we see are multimodal embedding models.

And the job of these models is essentially to take, uh, multiple types of data as input and generate embeddings for them so that all of these diverse data types can be searched and retrieved together using techniques like vector search, hybrid search, graph-based retrieval, whatever retrieval mechanism you want.

And the other class of models is multimodal LLMs, which can, um, be... DeepSeek does that at this point, Claude, uh, OpenAI, um, like ChatGPT has a voice mode. So- The job of these LLMs is to take all of these different data types as input and also generate, uh, outputs in these different data formats.

Now, if you give a multimodal LLM tools to search through multimodal data and use its reasoning capabilities to make sense of this information and to solve, uh, complex problems, what you have at your hands is a multimodal agent.

So let's build that. Enough talk. Let's actually talk about the agent that we are going to build today.

So we are gonna start with something simple. Uh, we are gonna,

um, remove as many abstractions as possible, uh, start with very simple objectives, uh, and build an agent from scratch. You get a really good understanding of what it really takes to, uh, build a multimodal agent in practice.

So our agent has two simple objectives, the first one being answer questions about a large corpus of documents, and then also given a chart or diagram, help the user, uh, make sense of it by explaining and analyzing that, uh, figure.

I do this all the time when I'm reading research papers. I'll just take a screenshot, pass it to Claude, and be like, "Explain that equation," especially with all those, like, mathematical symbols and whatnot.

Sounds pretty reasonable, easy. Not quite. There's a small catch, and the catch is we want to search our documents with mixed modalities. So in our case, our corpus is going to be, uh, documents that have text interleaved with things like images and tables.

And that complicates things because, uh, retrieving the right information from mixed modality documents is not a trivial problem.

The challenge lies in actually preparing the corpus of documents for search and retrieval. So typically, for text-based documents, if you've built a RAG application, you chunk up those documents, embed those chunks, and then, uh, retrieve relevant chunks to pass as context to an LLM.

But you can't really do this when you have images and tables in your documents. And one way to do this, there's so many tools out in the market. There's like LlamaParse, Unstructured, that use, uh, vision transformers or object recognition models to first identify and extract the different elements.

Like they'll extract text, images, tables separately. Then you chunk the text as usual, but you summarize the images and tables instead, and then basically convert everything to the text domain by, uh, creating embeddings of the text chunks and summaries using a text embedding model.

I know that's already a mouthful, but you'll see how to simplify this process using a new type of models. I'll just get to that in just a little bit.

Another technique similar to the previous one is that you still extract the text and non-text elements, you chunk the text, but instead of summarizing the images and tables, you would embed all of these, like the text chunks, images, and tables using a multimodal embedding model because, uh, it has the capacity to, um, understand and embed all of

these different data types. I already see some, some of you losing me because these, uh, pro-data processing pipelines are pretty complex, and they ha- they come with their own limitations, right?

They sound promising, but they have, um, mainly these two limitations. So the first one is that they face the same drawbacks, uh, that you see with chunking. So to me, the biggest problem with chunking is, uh, the loss of context at the chunk boundaries, which is why techniques like parent document retrieval, metadata pre-filtering are becoming popular, where

you add back context that was lost during chunking at either retrieval or generation time. Also notice how complex these, uh, processing pipelines were, right? You need an object recognition model, uh, to extract the elements.

Another, potentially another LLM call to, um, actually summarize these elements in addition to chunking and embedding, which is already one too many steps.

Another limitation with a lot of, uh, multimodal embedding models lies in the architecture of, uh, the models themselves. So until recently, the architecture of most multimodal embedding models, at least for text and images, was based on OpenAI's CLIP model.

Um, and what happens in this architecture is text and images are, um, passed through separate networks for generating the embeddings of these data types.

And this results in something, uh, we call a modality gap, where irrelevant items of the same modality end up, uh, close to each other rather than, uh, relevant items of different modalities.

So in a CLIP model, for example, text vectors of irrelevant text might appear close together in vector space rather than, uh, text and images corresponding to related subjects, and that's a problem.

But this has changed with the advent of, uh, vision language model or VLM-based, uh, architectures. So in this architecture, both moda-modalities are vectorized using the same encoder, and this ensures that both text and visual features are treated as part of a more unified representation than as distinct components.

So with these models, all you really need is a screenshot of documents containing whether it's purely images, purely text, or an, a combination of text, images, tables, et cetera.

And this is what, uh, because of that unified architecture, it ensures that the contextual relationships between, uh, text and visual data is preserved. So as you can imagine, this greatly simplifies the data processing pipeline for multimodal data and also ensures that you get, uh, better retrieval quality, uh, because you're no longer separating these texts and images.

So basically, given a document containing a combination of text and images, you simply take a screenshot of it, pass it through a multimodal embedding model, and the embedding that you get from that makes this data ready for retrieval.

Pretty straightforward process there. So let's quickly look at some of the... how some of the key features of the agent that we are going to build is work, and then we can go implement these in code.

So let's talk about the dep- data preparation pipeline for, uh, the corpus of documents that our agent is going to use to answer questions. So, like I mentioned, the first thing we are going to do is, um, for each document in our corpus, we are gonna convert that into a set of screenshots, and in our case, each,

uh, screenshot is going to represent a page in the document.

We'll then store the screenshots locally, but if you were to do this in production, then you might want to store them to some form of blob storage, like S3, uh, Google Cloud Storage, whatever, uh, your preferred cloud provider is.

And we'll also note the path to where the image is stored,

and then store this as metadata along with the embeddings of the screenshots generated using a multimodal embedding model, and store these into a vector database.

So in our lab, we'll use the latest multimodal, uh, embedding model from Voyage AI, and we'll use MongoDB as a vector database, because I work at MongoDB. [laughs] Uh, not that... [laughs]

So one thing to... important thing to note here that we are not storing the raw screenshots. Yes.

Yeah, one quick question.

Yeah.

Um, so you are retrieving the documents into screenshots. There are screenshots or you're picking up the screenshots from your document and sending into the...

So, okay, so say I have a PDF containing multiple pages. Each page in the PDF, I'm going to take a screenshot of it, and each screenshot is going to be saved separately in blob storage, and references will be stored as metadata along with embeddings in the vector database.

Okay. And why do you have to take the screenshot of each page? Is it to save the space or?

Uh, so like I showed you two methods before, right? Like, to be able to use the image and table data, uh, for reasoning, you need to, uh, be able to retrieve that document that contains all of that together.

And the reason I'm taking a screenshot is to preserve, like the continuity between the text elements and the image elements, so all of them can be retrieved together as context.

So this is, is worse than chunk- chunks, uh-

What's that?

I mean, a screenshot for each page-

Mm-hmm

... loses the context of, of the whole, I mean, document.

But then with chunking it's worse because you have like a very, uh... usually you're keeping like two paragraphs together or something. So, like slightly better, you would alway... of course, maybe want to augment this method with metadata pre-filtering, other methods that you use for traditional chunking.

But I still think one page is better than two paragraphs or a small paragraph of text. Yeah. Do you want overlapping segments here as well, just like you do with text chunking?

Uh, yes. Yeah. So when you... So here, we'll take screenshots of like distinct pages, but, uh, if you want that continuity, you might want to like keep some overlap as well.

Yeah.

How is this-

Good point

... better than giving it the full PDF at once?

Uh, because LLMs have like that lost in the middle problem. So if you give it... Like just because you have a large context window doesn't mean you should flood it, because they still have the problem of searching through like the full...

a large document to find the right, uh, information. So you're trying to-

Is the relation between these different screens shortly maintained after the

Uh-

... after the

Yeah. I think as, as she was pointing out, either you'd have to like, uh, maybe structure it a little differently to have some overlap or store some additional metadata to maintain that continuity, like page numbers or...

And whenever you're retrieving a page, you could do something like retrieve the previous two, next two pages, things like that. Yeah.

Thank you.

Any more questions?

Yes.

Yeah.

Uh, you talked about Voyage Multimodal 3. Like why-

Mm-hmm

... why that one versus others?

Uh, any VLM based model really. Like the whole point is to show you that clip-based models have that. I'm not using a clip-based model because it has a modality ga- uh, yeah, modality gap.

But any VLM based model is very good. Yeah. So this works for text and image. Yeah. Are there more like video- Right. Yeah ... voice? Like, uh, yep, 100%.

So this doesn't really deal with that, but essentially you could, um, extend this concept to different modalities. It's just I typically don't see like video or audio occurring with text.

Like I just chose this because, uh, like images, uh, figures typically, uh, occur with text, but... So yeah, screenshots might not apply to other modalities. There are different ways to handle it.

Today we are only focusing on images and text. Uh, okay, two more questions. [laughs] Okay. Yeah.

So VLM, um, I, I guess this, uh, is different from the large language model, right? So it has, I guess, smaller parameters than large language model, then-

Yeah

... performance is a bit less, I guess. Is it-

Um, it's just, it's still a large-

It depends on language to language.

Mm-hmm.

For example, like we make an embedding from the VLM-

Mm-hmm

... text, and then if you make an embedding from the, um, uh, let's say, um, Ara 002, which is bigger parameters, then performance probably, accuracy probably less.

VLMs tend to get pretty big too. So they're basically-

Oh, okay

... still large models. They just can handle like images and text, but they're still gr- pretty sizable models. [laughs] Yeah. And I could point you to s- to some benchmarks that show that they're still good at even purely text or purely image data, and then a combination of both.

How to run it on local?

What's that?

How to run it, uh, run in the local machine, I guess.

Um, like you'd find a model that works with your hardware specifications, just like a, an LLM, right? Like, not all LLMs can be run on your machine. So it would be similar to those.

Yeah Um, no, that was one

Let's skip one

Okay. One more.

Yeah, so in this case, you're using multimodal retrieval, right?

Mm-hmm.

It is true that your image and text is strongly aligned to the modalities. But what if you have a modality that's really weakly aligned to the time series, right?

So which means your latent space, they're not really close. How do you handle those?

Sorry, can you say that again?

If you have a modality that's, kind of, not strongly aligned with the rest of the modalities, like for the whole time series, right?

Mm-hmm.

If you embed it into the same latent space, they are not, like, really close to each other.

Yeah.

So in those situations, how do you handle it?

Um, so like time series data with text? Like, I'm, I'm trying to understand, like, a situation where you would have totally disparate-

So it could be like you have like a text, um, image and a time series too.

Mm-hmm.

And that time series data may not be really aligned with the rest.

Yeah.

And that means when you retrieve them-

Yeah

... they don't really go very well. So how do you handle those?

Yeah, I think time series data, typically you don't even, like, use embeddings for it. It's to... It's just like you treat them like any other features like you would for traditional ML models.

You definitely want like a different, uh, retrieval strategy for those. It'd be hard to put them in the same, uh, yeah, vector space as text and images. So you might need to work with like different retrieval methodologies.

Okay.

Yeah. All right. Uh, I'm gonna move, uh, forward here very... Like, in a few minutes, we will hit the hands-on portion. So if we have more questions, just, um, call out to our team, and we'll take more questions then.

Cool? All right. Um, okay. Okay, let's quickly talk about the workflow of our agent. We looked at a random example before, but let's talk about the agent you are buil- you're going to build.

So query comes in, agent forwards the query to a multimodal LLM. So note we are going to use a multimodal embedding model for retrieval, but we also need a multimodal LLM.

We are going to use, I think, Gemini 2.0 Flash Experimental, some long name. Uh, but yeah, basically we need that LLM because once we give it like, um, that interleaved document with text and images, we need, uh, an LLM that can make sense of both these modalities.

So that's why I'm using that LLM. It has, uh, just one tool, which is a vector search tool to, uh, retrieve those multimodal documents and also its past interactions and memory.

So based on the query, the LLM can decide to call the vector search tool, and if it does that, it'll return the name of the tool and the arguments, um, to use to call the tool.

Again, the agent has code to actually call the tool, so it calls the vector search tool. And typically, if you're working with text-based data, you get the, uh, documents back directly from, uh, vector search.

But in this case, what we are gonna get back is references to the screenshots. Remember, we didn't store those in the vector database. Those are in our local or blob storage.

So then our agent needs to have that additional step of using those image references to actually get the screenshots from blob storage, and then it's going to pass those images along with the original user query and, uh, any past conversational history to the multimodal LLM.

So each time an LLM call is made, whether it's to determine what tools to call or generate the final answer, uh, the images are also going to be passed along with the query and conversation history to the LLM.

Then it generates an answer, and that gets, uh, returned back to the user.

And finally, depending on the query, the LLM might also decide that it doesn't need to call a tool. So for example, if the user is simply asking like, "Hey, summarize this image," it might not need to, uh, call a tool.

So in that case, it'll say, "I don't need to call tools." It'll simply generate an answer, and that gets forwarded to the user.

Um, and the final thing, um, let's talk about the memory management mechanism for our agent because this is important for it to actually have, uh, coherent multi-turn conversations with the user.

So like I mentioned before, we'll be implementing short-term memory for the agent, and the way this works is each user query is associated with a session ID, just as some identifier to distinguish between different conversations.

So given a user query, we obtain its session or conversation ID, and we query a database consisting of previous turns in the conversation to get that, uh, chat history for that session.

And each time, again, in addition to, like, the context, we also pass in that, uh, chat history just so the LLM can use that as additional context to determine if it need, it even needs to call tools or not.

And then when the LLM generates a response, the other thing that happens is we add, uh, this current response and the current, um, query back to the database to add on to, uh, the history for that session.

Now, you can also log, uh, tool call, their outcomes, any, um, reasoning traces from the LLM, but at a minimum you at least want to be logging, uh, the LLM's response and the user queries themselves.

And finally, that is enough talking from me. You... I'll be quiet for the rest of, uh, the workshop. We have about forty-five or s- forty-five-ish minutes. So head over to that link, uh, to access the hands-on lab.

Recommend running that on your laptop. So instead of QR, that QR code, uh, actually type in that URL. This should take you to a GitHub repo. Uh, follow the instructions in the Read Me to get set up for the lab.

That should take about ten minutes, and then you have two options. Um, you can either... There are two notebooks in there. One is called lab.ipynb. And if you actually want to, if you're in the mood to actually write code right now, that's the notebook you'll be using.

You'll have, you'll see reference documentation in line in the notebook indicated by that books emoji that tells you, "Use this documentation, fill in your code." You can do that.

If that sounds too daunting, there's also a notebook called, called solutions.ipynb that has all the code prefilled. So you can just run through that notebook, read the comments to get an understanding of, uh, how the agent works.

But whichever option you use, I'm here, my team is here. Just call on us to, uh, if you have any questions. And yeah, for anyone actually filling in the code, you can also refer to the solutions.

Don't get too frustrated if you get stuck. Uh, we don't want that. All right. So

I'm shutting up now. Um, let's go ahead and build that agent. [clapping] [outro music]