Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Hi. Hi, everyone. Uh, thank you for coming to this session. Uh, we're-- It's gonna be a, a session about agent harnesses. We don't have a lot of time. It's only one-hour session, so, um, hopefully we have time to talk about a lot of things that, that you guys are interested in. And the, the session essentially is gonna be, uh, like, me talking, like, ten, fifteen minutes about the topics that we're gonna talk about and solve in the workshop. You have the workshop instructions, uh, in a QR code or in a

website, and if you don't wanna listen to me, you think I'm boring or something, you can start, uh, on your own on the workshop as I'm speaking. But essentially, this i- this is what's, uh, happening on the website. You just, um, sign up with your GitHub, uh, handle, and then you will automatically receive an invitation to the repository. The workshop is hosted on Codespaces, so you can just, like, create a codespace out of that, uh, and you will receive an invitation

automatically to get access to the workshop itself. And you just have to create a, a codespace over there, uh, open in GitHub Codespaces, and you're good to go and, and, and begin with the workshop. Uh, so I'll just put the QR code, uh, one more time for those of you who were-- uh, who, who wanna take that, and I'll, I'll get started.

Okay. Perfect. Uh, if you have any questions also, my colleague Richmond is over there. He's gonna go through, um, you know, everyone's, uh, table. If you have any questions, just raise your hand and he can help out as well.

Uh, yeah. There you go. Also the website, I mean, you can go to the website workshopwaitingroom.com. I paid too much money for that domain, so, uh, use it.

Okay. So who am I? My name is Nacho, Nacho Martinez. I am an AI developer advocate and a data scientist by heart, and if you're interested in anything about me, just find me on YouTube. I have podcasts. I have workshops. I have e-everything. Not gonna go too much into myself. Um, just one thing though, uh, we re-recently released a course with Andrew Ng and DeepLearningAI about, uh, building adaptive AI agents, which is one of the

subcomponents that we're gonna talk about today, and, you know, if you're particularly interested in the things that you can do on a model to make it better and adapt over time, learn over time, just check out that course. It's a free course, uh, a one-hour course that you can check out on, on Google. So what are we gonna go, uh, through today? Like I said, we don't have a lot of time, so I'm gonna try to be quick. Um, but these are the things that we're gonna talk about. First of all, what is the agent stack?

Uh, what are the components of an agent stack? What are the different shapes that an AI application can take? What an AI agent is, and what is the agent harness? What is the main motivation of building or working around agent harnesses rather than working with AI agents or with an LLM model directly? And, you know, all the subcomponents of an agent harness, like for instance, memory, context engineering, memory engineering, and all these things. So because they told me that there's gonna be a

lot of people from, uh, financial, uh, like a lot of people that work in Wall Street, uh, I adapted this workshop to be around a bank, right? So the idea is that the main premise of the workshop is this. We are hearing lots of the, uh, nowadays, like, OpenAI is launching ten thousand agents or five thousand agents to solve this problem, ten thousand agents to solve this mathematical problem that wasn't solved, like, uh, in a hundred years, two hundred years. The thing is that these agents

are autonomous. They don't have any human in the loop until they arrive to, uh, to the conclusion or the answers, right? So the agents themselves don't have the capacity to do this. Why? Because, I mean, the context window is limited. The memory components are something that a, a large language model doesn't have inherently. So all these things we need to implement ourselves to make these agents able to solve all these problems for ten hours, a hundred hours, uh, autonomously, right? So the idea is that a lot of

people are making focus on, okay, we have these frontier models. But my idea is that, okay, we don't need to focus so much on the frontier models. We can use them, but let's work on the harness, which is the interesting part and the thing that you can actually customize, um, more and more. So on this application, the idea is that it's a, um, medium-sized bank that has lots of, uh, you know, s-suspicious transactions, uh, sometimes, and they are categorized into five

different types. So for example, if you get, like, a million-dollar transaction that's coming into your bank and automatically and magically, uh, for three days you're making, like, fifty-thousand-dollar withdrawals on an ATM, that's suspicious, right? Because why would you, like, put one million dollars in and then just, like, distribute, uh, like that? So the idea is that we have this medium bi-- uh, medium-sized bank that has lots of transactions, and these transactions, they need to be kind

of autonomously decided or at least flagged so that a human that actually works in the bank needs to spend less time actually looking at these things, you know? So let's go to the theory a little bit. What is the agent stack, right? When we talk with an AI agent, we typically are talking with the application layer, right? Uh, what we interact with, either, um, an interface, a chatbot, um, a mobile application, et cetera.

Then we have the model itself, which is the reasoning core, the, the-- you know, we don't get any say in what the decisions are. Uh, and then we have also the infrastructure that supports it all and then the compute. But what we are gonna focus about today is the data part of it. And the data is very important because everything that we can manipulate and work around is on this part. The other parts, we get less control over them, right? Especially if we're using a, like, a

third-party, uh, model that isn't ours. So the data layer is what we focus about, uh, today, right? First of all, we have the agent loop, and this is arguably one of the most important parts of an AI agent. Uh, the agent loop is in charge of essentially driving the whole thing, uh, and it's customizable as well. And inside the data, you also have like a gateway and MCP so that you can connect it to third-party external data sources. You

also have the memory layer, which we will talk about later, and essentially also some context engineering, uh, parts of it that let you automatically summarize and compact and shrink the working context window of a model. And finally, also some tools and skills, uh, which we can do some interesting and intelligent stuff so that we can essentially keep the context window limited. This is one of the main ideas of the workshop.

Now, typically, the applications that we interact with, they are either a combination of these four, right? LLM chatbots, which are very easy. Lots of people just create an AI application, create a chatbot so that you can talk with it. You also have RAG applications, which is essentially like a glorified, uh, vector search. And then you also have LLM-driven workflows and AI agents. And typically, what most of us use... Uh, by the way, raise of hands, who doesn't use AI

agents like Claude Code, et cetera? Of course. Uh, so Cl- Claude Code, Codex, et cetera, all these systems are a combination of three and four, which is, um, you know, a combination of LLM-driven workflows that provide automation and AI agents that provide autonomy. So it's a, a combination of automation and autonomy, and these are called agentic systems. The better systems like Claude Code, Codex, Pi, uh, OpenClaw, Hermes Agent,

et cetera, they're all a combination of these two types. And most of the engineers that I talk to, and this is, uh, important, is that we use these systems every day, right? But we stop, or a lot of people-- not a lot of people think about these problems when building their AI applications. So you might have an AI application, but then you don't consume it like this, and this is something that we're gonna focus on today because I think this is the better way of consuming, um, AI applications, not only through

chatbots, not only through RAG applications, but also through a combination of automation and autonomy. So what is an AI agent? For those of you, uh, who don't know, it's, uh, like a computational entity that is powered by an LLM, by reasoning, kind of like us humans, we have our brain. It's augmented by a database for memory, also like our brain. It's extended through tools for actions. So the equivalent of a human would be

like we have our extremities, and we can interact with the world, just like an AI agent can interact with an operating system through the tools that we give it, that provide it. And it's also grounded in inputs. So for exa-- for instance, it would be like our senses for a human, our eyes, et cetera, that let us per- perceive information. And the context, like our, our attention, is also limited, like our agents. We can't watch seven movies at the same time. Like, we will not understand any of the movies. We will not

focus on any of them. So the context window for humans is similar to AI agents, and kind of we try to assimilate or, you know, have some of these notions be closer to our human experience, and this is something that we'll, we'll work on today. So the reasoning part of it, uh, is provided by the frontier labs, and we don't get a lot of control over that. And then we also have the things that we do have a little bit more control over, which

are the memory, the tool uses, and the perception, right? And this is the, the harness itself. The harness is essentially all the scaffolding that you need to do to build, uh, these three components, which is the, the tool use, the memory, and the perception. And we're not gonna focus too much on the model because, like I said, we don't have a lot of possibilities on what to do, right? So the harness

is actually one of the most interesting topics in AI. I think it's the most interesting topic in AI nowadays because these harnesses actually control everything that the AI agent can do. You give it a different configuration than the agent, the same LLM frontier AI model that you're consuming becomes better or worse. Sometimes, you know, it's been proven that by just changing the harness, the same model performs better or worse for specific tasks. So it's really, really

important to make focus on the harnesses, and not a lot of people are doing this. And like you said, in a harness, there are, you know, seven different or eight different, uh, components if you consider continual learning as well. But there are lots of decisions that you need to make, right? You have the model. You have the storage, like where do you put the data. Uh, you have some encoding and retrieval components, like how do I make the distance between two vectors to actually retrieve the data properly? Uh, what do I

store in my memory? Do I think of it as short-term memory plus long-term memory, kind of like humans have? Do I think of it in different, uh, compartments or different types of memory? Uh, also the semantic layer, which is a part that's really, um, interesting because as humans, we are here, and we have some context that we have never said to each other, right? But we are sharing this. For instance, I'm a speaker, and you are listening. You are not singing right

now. Why are you not singing? Because it's not appropriate in the semantics of the context that we are at right now, right? So the model itself also has access to the semantic layer, which talks a little bit about all the things that haven't been said to the model, but they are assumed. Also, the agent loop, which is, like, all the things that drive and automatically and autonomously make the agent work all over, uh, again, and this, uh, is, like, one of the most important

parts. And also context engineering. And we'll focus on each of them, but I will not go that... Uh, w- I will not deep dive into them because we'll actually do that in the workshop. So just for reference, um, there was this big discussion around files versus databases. Like, a lot of people were saying, "We need to use file for everything." Like, you just put everything into a file, and then, like, magically, one month later, the agent becomes better. Some other people are database maximalists, and they

wanna put everything into a database, right? So this dilemma is still going on, and I would say that, um... Well, let, let me give you an example. Like, can t- can anyone tell me, in this example that you see on the screen, can anyone tell me what is the workaround that agents use to solve, uh, this specific problem that you're seeing on screen? Multiple agents writing to the same file. How does

that work nowadays in AI agents? Any ideas?

Worktrees.

Worktrees. Exactly. Good. So worktrees is the current solution, right? And this is a, a thing or a workaround because files have a, an inherent problem in themselves, which is that they, they, like, if multiple writes are coming into the same file, then you have problems with consistency. Like, which write came first? If someone is reading, what do I do with this person that read the file before I modified it, et cetera. Lots of problems, right? So for some things, I will definitely tell you, like,

files are great. Files are great at storing lots of unstructured data. Database on, on the other hand, is also used for some other types of use cases. But my solution is that for some cases, some of these, uh, solutions work with files. For some other cases, they work with databases. So we actually, um, need to reconsider some of these things that are going on, right? Um, and the worktree, uh, so- solution, or the workaround, has been already solved by

databases 30 years ago. And we're trying to reinvent the wheel with some of this, and we don't actually need to reinvent anything because there is an actual solution to this. For example, if you're trying to write to a file, um, at the same time, multiple agents, if you have transactional consistency or ACID consistency like atomic, consistent, isolated, and durable, um, writes into the file, you will not get this issue, right? So

files are good for some things, and databases are good for, for other things. We don't have to go into only one of them, right? So in the agent harness, just consider this as an i- uh, as a thought.

Uh, another component is the... all the memory components, right? And this comprises all the mechanics and the, the systems that are created around an AI agent to make it use the information, um, and not only use this information but also be able to refine it and recall it appropriately when it's time, right? And agent memory, you... As the engineers, you get to choose, like, what are the things in your agent

memory layer that you're gonna consider. Some people like the short-term versus long-term memory, and in this case, this is part two. We have some short-term memory components. For example, if we are talking with Claude or with Codex, we have short-term memory in the sense that the context window itself is short-term memory. Anything that disappears after the con... Like, it disappears after the conversation, right? Most things, uh, in the context

window just vanish. Then you launch another conversation, and it starts from zero. That's short-term memory. It's very limited. We cannot put everything into it. And then we also have long-term memory. And inside the long-term memory, we also have some categories that we propose. Uh, for example, episodic memory would be all the things that happened, all the episodes of our conversations, all the tool logs, the previous conversations, um, summaries of previous conversations that you've had with the agent, et cetera. They

all go into this episodic long-term memory ca- category. You also have procedures or procedural memory, which are all the things like skills, workflows, which are like a sequence of skills that are being run, et cetera. All of those go into procedural memory. And also, we have semantic memory, which is all the unsaid, like I said. The agent remembers who you are, what are your preferences, what's your, you know, favorite framework, et cetera. All these things go into semantic memory.

So the problem, the main issue here, why cannot we put everything into the short-term memory? Because it's limited. And it's not only limited, but actually, the bigger the attention matrix is, and the attention matrix is essentially the, the context window, the bigger it is, the worse it performs. And this happens for every model. There are lots of studies that have shown that- If you start at zero tokens on the context window, the problem is being

solved much more accurately than if you have two hundred and fifty-six thousand tokens already. And the limit is a million right now. We're at a million because of the KV cache. All these things need to be considered, and just know that even if the advertised limit stops at, uh, two fifty-six K or one million, if you keep going, the, like the performance goes horribly. So the smaller the context window is, the better it will solve a problem. And this

applies for every type of problem that you can think of, uh, in humanity, right? And there are two types of things, uh, that tackle this. One is context summarization, which you just ask it, "Hey, just generate a summary, a short summary for me with this, uh, with all the things that have been said in the context window," and then you get the summary, and then you start off from the summary the next conversation. That is fine. Uh, another one is compaction, and that's kind of similar, but in compaction, you're just, like, putting chunks of the conversation into the

database or into a file, and then you just put it with an ID. And if the AI agent needs it later, it has the ID. It just pulls from the ID with an offset, kind of like, uh, reading from a file, right? Some of the... These are some of the problems that happen in context, uh, context engineering. And then we have the agent loop, which is the driver or the, the car that, that drives all the components that we have talked about. We have a context assembly part that pulls

from the skills, pulls from the tools, pulls from the previous conversation, from the files, from the database, all of these things. Just brings in, uh, brings into the context window dynamically. Then it does some reasoning. It makes a, an API call to OpenAI or whichever model you're talking to and decides on what needs to be done. And this happens, you know, automatically and infinitely if you want. In my case, in this agent loop that, that, uh, we prepared for the workshop, it has a

maximum patience because I don't want to run out of, uh, OpenAI credits or any, any credits that I, that I have. But, uh, yeah, the, the maximum number of iterations in our case will be eight, and a maximum of sixt- sixty seconds of reasoning for each one of the iterations, right? Um, but this is completely configurable. Like I said, us engineers, we have to say in what goes on into the agent loop, right? And also the toolbox pattern, the skillbox pattern, what we

propose for this, for storing tools and skills, which are hundreds... I don't know about you, but I have, like, a hundred skills already installed. I'm not putting a hundred skills at the same time into the context window because each skill is, like, three thousand, five thousand, uh, tokens. So that would automatically, like, go over the advertised limit in no time. I, I would have no space for my actual conversation with the agent. So how do you fix that? Well, there are different ways, but what we propose is that you

vectorize, you create vector embeddings out of the tools and out of the skills. And through those vector embeddings, once the agent loop... Or let me go back. Once the agent loop wants to retrieve some useful skills or tools, you're gonna do vector similarity search against this vector store that you have. You're gonna pull the most relevant skill or tool to solve it. You solve the problem, and then you offload it again. And that's one of the ways in which you, uh, keep the context window relevant in this, in this

sense. And, you know, using skills and using tools is a very, very, uh, common pattern. Like, everyone's using tools like a bash command or a change directory. All these are specific tools through bash, right? So we're using tools and skills constantly. This is, for some cases, more than fifty percent of the context window that's being used, right? So I just want you to be mindful of the vast amount of context window size that is actually being kept

by the skills and by the tools.

Um, we also have this idea called, uh, skill distillation, where you take previous conversations, runs, and, you know, actual, like, conversations that you've had with an AI agent, and then you try to create something meaningful, meaningful out of that. How do you do that? Well, you can do it in several ways, but one of the easiest ways that you can do it is just, like, create a skill.md and put it into the same skill box that you had everything before. And this is one of the ways,

like, uh, Hermes Agent, one of the innovators in this sense, was creating skills that are particularly good to solve specific, uh, specific problems. Let's say that you're working on a front end. Well, if you're working on front end, you might wanna create a skill that has worked very well for you with all your preferences. "Hey, I like this coloring style," et cetera. And the next time you work on the front end, since you have created vector embeddings out of this, you can just, like... The agent will automatically choose

the more relevant skill. So if you just mention, "Hey, I like this coloring style that I had in this other skill," it's automatically gonna know that you're referring to the new skill because you used the word, uh, coloring style, for ex- for instance, right? So skill bo- like, skill distillation is one of the techniques that, that you can use to improve the, the learning, the learning of a model over time. And it's the same. Uh, this is, uh, related to what I, what I was saying,

is that you have dynamic skill discovery in the sense that the agent loop, when assembling or building the context, will pull the right skills and the right tools when it needs. And, you know, some people ask, uh, like, "How does a frozen model get better?" And what I mean by frozen is that we, we don't get any control over the frontier models, right? Uh, but we do get control over all the data c- the data layer of it. Right? And some people say, "Hey, let's

modify the weights of the model." Some other people say, "Let's create an embedding model that works better for this specific problem, and then I'll worry about, you know, just retrieving the better embeddings there, uh, et cetera." So there are different ways to do that. But the idea is that this is the main component of continual learning, and continual learning is one part of, of, of the things that you can do to make a mo- a model better, and there are three categories, right? You either work on the

parametric memory. It's called parametric because it modifies the parameters of the weights of the neural network. You also have representational memory, which is what it works with, uh, embeddings, et cetera. And you also have nonparametric memory, which is everything that we have spoken about today until this point. But, um, if you're interested in any of these three, we have focused on the third one. You also have the other two, and I will not go into the, uh, into those two. There are lots of techniques

like, um, you know, creating low-rank adaptation, uh, fine-tuning a model, uh, creating embeddings, uh, et cetera. And all these things you can find in that course that I re- I was referring to before, which focuses on the three categories. So if you're interested, just take out, uh, take that course if, uh... or chat, chat with us after, after the session. So after the session, we'll be in the event for the, for the rest of the event too. Right? And last thing I wanna mention is that

all of the things that happen during a chat completions API call to any, uh, OpenAI model or anything that you're using, is that there is an inherent, uh, instruction hierarchy. What do I mean by instruction hierarchy? There is a different weight on the things that are given to the model. For instance, what we use to interact with is the user, right? The request. In this sense, this is the third most important thing that we can pass to a model. The most important one is the

system prompt, right? Everyone knows about this, uh, the system prompt concept. Um, and the system prompt is fine, but it, it cannot hold everything because otherwise things would not have a hierarchy in itself. Uh, you need to actually put some of the things in the system prompt, some of the things in the user prompt, and all the tools and the skills that... the things that are made available, they have this hierarchy. And this is good because it, it's, it actually, um, lets us or lets the model focus

more on some things than others, and we'll also go through that in the workshop. So just for reference, uh, this is, uh, the, the UI of the workshop. So if you're running it in Codespaces already, you'll see that you have, uh, this deployed. I will go through, through it, uh, with you. But this is essentially the way in which I envision someone that works in AML, like, um, a, a fraud detection system in a, in a bank, interacts with an autonomous agent.

The autonomous agent is running in the background twenty four/seven. It doesn't sleep. It doesn't need any of the humanly, uh, necessities that we have, right? It's connected through a harness to the, to a database, and it has all the functionalities that we have mentioned. But on top of that, uh, it's autonomously making decisions for the company even when, you know, we're offline. It's not ruining people's lives by just, like, saying, "Hey, this, this guy is suspicious.

Let's just, like, block the whole bank and everything," you know? Uh, because that would ruin, ruin the person's life. But rather it's making these decisions and just, like, printing them so that a human can actually inspect them later on. But it's gonna save a lot of time during the, you know, the discovery process, et cetera. So by the end of the, uh, of the workshop, if you complete it, uh, you're gonna be able to have an agent that connects to a database that is actually

working as if it was, uh, any, any company, and it's gonna do autonomous analysis and decision making that then a human is gonna take and interact with. So these are all the, the components that we have talked about, about the harness. And without further ado, I will, uh, put the QR code just again or workshopwaitingroom.com. Um, and if you wanna check out any of the other workshops that we have, uh, you can

just check out this GitHub repo. We're making a lot of, uh, progress and effort into all the things that go in there. Everything from workshops, sessions, uh, notebooks, so you can get started on different topics like agent harnesses, et cetera. So just check it out. Uh, and let me, let me go to my Firefox and show you how this application works. So first of all, let me... just, uh, for those of you who didn't

receive an invitation, if you go to workshopwaitingroom.com, you will get an invitation via the GitHub API with your, uh, user to this repository. So if you just go here and then open on GitHub Codespaces, it's gonna ask you, uh, to create a codespace, and of course, I'm paying for it, so don't worry about that. Uh, just be careful with, um, with the hardware that you deploy because it's expensive.

But, uh, the idea is that you will get a, a deployment like this with a couple of notebooks, and I'll show you the workshop itself. And then my idea is for me to step down in a couple minutes, and then we'll go through the, you know, people who have questions, et cetera, and then we'll solve it together. But it's automa- automatically also deploys this application, which, uh, essentially is tracking all the things... and this is, of course, uh, fake data. Don't worry, I'm not...

Exposing anyone here. But it's essentially coming up with new transactions. Some of them are fine, some of them are, uh, problematic. So the autonomous agent, which is our Groq model running on OCI, it's essentially making autonomous decisions, and some of them it's saying, "Hey, this one is suspicious. Let's add it to the list." And then afterwards, uh, you also get like a, a way to visualize the context window of the agent here. Um, you can also

check out all the data that's already in the database, uh, here, uh, like all the customers, et cetera. And also the autonomous way in which the agent is actually, uh, gonna make an analysis of all this, um, of, of all this in real time, right? So it's gonna run through the tool calls that we give, uh... that we have available in our tool, um, in our toolbox. Uh, yeah, it's... Sorry, it's a little small. Um, you know, so it's gonna go

through the evidence, it's gonna search for knowledge in the knowledge spaces that we created through vector stores. It's gonna make a lot of tool calls, and finally it's gonna essentially go through the whole activity, uh, suspicious activity reports automatically. And here is just a fancy way of visualizing the 30 most recent, um, events that happened in the bank. But the workshop itself is this notebook, the notebook student.ipynb. And what I want you to do is

first of all just check out the different components that we have there, because we are implementing them, uh, here. And I'm gonna go through the first, uh, the first to-do. There are a couple of things that you need to do on the workshop just to confirm that, that you know, uh, what you're doing. So first of all, let me just go ahead and show you, uh, the first part of the workshop. Right? Uh, you have this Groq model 4.3, uh, which is running on, on Cla- Oracle Cloud.

And, uh, let's, uh, connect to the database of course.

And after... Sorry, let me restart the kernel.

Let me...

run this cell. Restart. But yeah, uh, let, let us know if you have any questions or issues, uh, when getting into the Codespace, and then I'll step down, um, and do this with you and answer any questions.

Any questions, by the way, so far? Okay.

Yes?

Uh, the, the repo?

Yeah.

Yes. You can find it on workshopwaitingroom.com. And let me just put the QR code here, uh, this QR code if you wanna, uh, scan it. And this will automatically give you access to, to it via GitHub API. Okay, so let me re- uh, reframe the question for the people who are on YouTube. Um, so he's asking about what happens with the episodic memory and all these memory components. Does it get it confused

with previous conversations with a, with an agent? Well, it depends on how much, uh, weight you give it, and this is why I was mentioning the instruction hierarchy as well. You might wanna put this... If it's a really, really important thing and you actually want it to pull from some place or the other place, uh, you can actually decide your memory layer architecture yourself, right? So you might wanna, for instance, in some cases where the episodic memory is really important and you actually wanna pull from previous

conversations, you might actually wanna vectorize that rather than just save it r- as raw text, right? Depends a lot on the, on the use case, I would say. In this case, it doesn't co- uh, it doesn't... There is no confliction there because we have a clear separation on the memory, uh, components of the architecture. Yes. So the, the skills can be added to, to the system prompt. Uh, they typically, um... Oh, sorry. They typically have a different... Uh, yeah. I- in the instruction hierarchy, they have the

least weight out of the four. Uh, if you were to think about it, it, um, it's typically here, so it has the least weight. I would think of it as a plug-and-play place in which you can pull things. And we are experimenting with this, uh, actually Richmond and I, because we actually wanna create a way in which you can plug into external or third party skill hubs and tool hubs. For instance, a collection of MCPs that you can give your agent a-

availability by pre-computing the embeddings of all of these things. And once you have these pre-computed embeddings, you can just like say, "Hey, do vector search against this third party, uh, group of skills or group of tools." But rather than thinking of it as being included... And by the way, this is of course included in every OpenAI or every chat completions API request that you're making to OpenAI or any other model. This is attached at the bottom. And this typically is

like 80% of it. Like 80% of the whole request is, uh, is, is tools and skills. But what, what happens or, like, why don't you get charged like five times more than what you're charged now is because, uh, OpenAI and all these systems, they have a semantic cache. And they put, uh... If 80% of it was the same as the previous request, it doesn't recompute it, it just goes into the semantic cache, and this allows you, and this allows all of us to have, um,

an, a way in which to consume the Frontier Labs models in an economical way. And of course it's still very expensive, but what I want you to understand is that it would be four or five times more expensive if we didn't have the semantic cache that just, um, puts the tools and the skills Into it. Because in the response, like remember that when you interact with, ah, interacting with a frontier AI model, you're essentially making the whole thing through an

HTTP request. And everything... Like this would be, for example, 10K or 5K as a system prompt. You're an AI assistant, and then you need to help me solve this, this, and that. You have the developer, like the specific app instructions, the user query, which is what we type in the chat, which is n- negligible, like te- 100 tokens, 1,000 tokens maybe. And then the tool outputs and the, the retrieved docs, which is 80% of the request, like 35 to 100,000 tokens. It's already

this in every request. Does that answer your question? One last question. Yes. Um- Okay. So his question was like, how do you manage the budget? And since there are so many things in the skill box and the toolbox patterns, like how do you essentially make this work, right? Um, we have a clear separation. The context window, we think of it as... Or think of it as an N-dimensional matrix that starts at zero, but it's never zero. It's closer to like 10,000.

I was reading a paper that was comparing the efficiency of each harness on how much they put into the first system prompt and the first, uh, completions API request. At the beginning, Pi... You know Pi, the harness? That's the most minimalistic and the most efficient, uh, harness because it includes about 6,000 tokens at the beginning on every request. But if you go to other harnesses, they have more like 30, 40,000 tokens. Um, how do we... We have a clear

separation. The context window, we just let the... E- everything goes around the agent loop, right? So let me go to the agent loop to show you, uh, one thing. The agent loop will control everything that it connects to, right? Um, so here, for example, the agent loop... Oh, I'm at the bottom. Hold on.

Uh, just as a reference, um,

the agent loop.

So here, right? Uh, the context assembly, when you're working with it, uh, you will essentially build... And you can go through this in the, in the notebook. It's explained in the notebook. But, um, the context assembly is one part that, first of all, we have a maximum number of defined, uh, tools or skills to retrieve. I think the defi- the default number is three. Uh, so every time a new request is coming, it's

determining... And this is done by the model itself. It's determining what is the thing to do. Then it's gonna assem- try to assemble the context with the things that it needs to perform, right? And one of these things is retrieving the three most relevant tools, the three most relevant skills, and we are not using a re-ranker, but what you could als- also do is like at the, at the end of, uh, the similarity search, you can also put an embedding model plus a re-ranker

that will take those three and tell you out of these three, this is the most important one out of the three. And you could actually have a better refinement out of that because you, you might not want to have a total of three skills loaded for every task. The task might be solvable by only one, right? So the idea is that you can put, uh, whichever strategy you want, just ta- uh, stick with it and, uh, stay consistent with it at... until the end. Uh, so the idea in this case, in the particular workshop,

is that we have a total number of, uh, vector searches that we're doing against one vector store, which holds the tools, w- another vector store that holds the skills, K equals three, and then out of the three, you can retrieve the most relevant one with a re-ranker. And then it's gonna dispatch the tools of course, and build the context window. But this is entirely the decision and the job of the agent loop. Yep. Okay. So what is the most, uh, important

or interesting way in which you, you can interact with the memory layer of- For your agent- I mean, the efficiency comes just from the number of entities that you create, right? So my first recommendation w- would be just like, keep it simple. Uh, so don't create too many abstractions or too many components of the memory layer. So for instance, try to experiment, first of all, with only short-term memory and long-term memory, and just have two structures, right? If you think, or if you have an agent

that considers that something that was in the short-term memory or in the context window is kept... is, is worth keeping, you promote it to the long-term memory, and you just put it somewhere as raw text. That's the easiest thing that you can do, right? Once you have it in text somewhere else, in a long-term memory or component or object, then you can take that, vectorize it, and then let it be explored by the agent loop in some way. So for

instance, here, you would just have like an extra thingy in the middle of observing and reasoning that lets the agent recall memories or something. You can call it like recall, recall long-term memory, and this is gonna make it, uh, be introspective before solving the problem once more. Uh, so it's gonna go to your vector store or wherever you have this long-term memory, uh, episode that you recorded in a previous conversation, and from

that you can let it see, okay, I'm gonna let it pull from it. If it considers it to be interesting or if the vector similar... like if the distance between the vector of the query and what I have in my long-term memory is relevant enough and the distance is small enough, then I will consider it into the context window. If it's too big or it's not relevant enough, then I will just discard it. So it will be like a different... uh, an additional abstraction between the observing

and the reasoning part. There are ways. We are not doing MCPs right here, but one of the benefits of, of using MCPs would be to solve, um... I would say that MCP is not really that, that relevant in this, in this case, um, but it depends on what you have in the database. So for example, if you have a database, you're interacting with a database like we are doing right here, having an MCP is interesting. But one of the main problems that I like to say to people, like what does an MCP solve,

especially in the database, because I'm a database guy, because I work in Oracle. Um, so one of the things that you can solve is letting AI agents make SQL queries, but by not hallucinating or trying to come up with a new SQL query every time. This is one of the bigger problems that, that AI agents have with SQL. SQL has like 15 times, uh, like 15 engines. Uh, like Postgres, you have Oracle Database, you have VSQL, you have like lots of different queries

that work on SQL, but they have specific hints or subtleties in the query that makes it not, not work well, right? So the main problem that MCP solves in this case would be, okay, I'm gonna pre-create some SQL queries, and then let the agents access those. So those queries are proven to work, um, rather than having an agent try to come up with a SQL query dynamically every time. Right? Mm-hmm. Yep, good question. Uh,

let me give you an example. So the most important thing would be the rapidness and that the meaning of the sentences and what the skill means itself, uh, rather than having it as text. And if you were to store this just as text in a SQL file, it would be able to... Sorry, in a, in a file, in the file system, right? In the operating system. Then you would have, uh, lots of ways in which an agent can do regular expression matching. So for

instance, all the search would basically be either keyword search or like k- word search based, uh, into that skill, which is fine. Um, but the bigger that gets, the less imp- like the, the, the harder it gets for an AI agent. First of all, regular expression matching like grep is inefficient for huge files, and sometimes if, uh, like the, the file system grows and grows linearly, the performance of the

regular expression matching decreases exponentially. Well, not e- exponentially, quadratically. Let's say just quadratically. But this is one of the problems, um, that it doesn't scale as well as the size of the files. Uh, another one would be that if you precompute the embeddings of a skill, you're actually also capturing the meaning in these embeddings, the hidden meaning that we as humans don't... cannot comprehend. But rather than just, uh, describing it in text, which is fine for

us to read, uh, it also has a precomputation of 384 minimum to 1,000-plus, uh, dimensions. And this embedding will actually hold, uh, things between each one of the tokens in the, in the, in the embedding matrix to be much more discoverable, for instance, by semantic search. And then this is what leads you to try hybrid search, which is a combination of semantic search, uh, which is embeddings,

plus keyword search. And, uh, and this is called RRF, uh, which is a type of hybrid retrieval. It performs much better because it will essentially just have both of them. Uh, and it will essentially... And this is, uh... We can... I, I can show you, uh, the example in the, in the workshop because I, I created this visuali- uh, visualization here that explains exactly what I mean. With hybrid search, right? Uh, you see? Yeah. So depending on the, on

the... This essentially is a plot that, uh, combines the search capability of keyword search plus semantic search, and this gap that you see at the middle is anything that's, that's plotted here when doing hybrid search, it will be more relevant than any of the other two. So, uh, as a summary, uh, the more... Like the more... You can have two different types of, of search. Each one has its advantages, but if you combine

them both, you will, uh, effectively do a little bit more compute, but then the quality of the retrieval will, will be better over time. Um, these are... Like most of this, I'm thinking of it from the AI agent perspective, so I'm not expecting anyone, uh, from this, uh, room to just like start building their own custom harnesses because we already have great, uh, solutions in the, in the market, right? We have Claude Code, we have Codex. Each one has their own, um, preference. By the way,

show of hands for who likes Pi.

Okay. Less than I expected by... But, uh, good. So, um, I would say that I'm thinking of it as from the AI agent perspective. This... What I want you to understand is that you can build custom harnesses for specific problems, and building custom harnesses doesn't to be, uh, uh, doesn't have to be as sophisticated as creating the new Claurk- the new Claude Code, and then you're running all of your work in that custom harness. Rather, let's simplify the

problems that you're trying to solve and then build custom harnesses that attach onto, onto that. So in this case, I'm thinking of, um, you know, if you're a consultant, you are my perfect, uh, use case, uh, or my perfect user for this workshop because you could be creating a custom harness that- ... is working fine for a financial agent and has all the things that it needs to a financial, uh, use case. But then if you have another type of customer or with different types of data, or with different types of constraints, if you wanna solve

the next, uh, Navier-Stokes problem or something, uh, you know, if... Those have different constraints. For e- for instance, the OpenAI problems, the, the mathematical problems, they need an infinite... an, an automatic compaction or a- automatic context summarization because these agents don't have any human in the loop in the middle. So they c- like, everything that goes around pro- solving these problems have to have, by default, automatics, uh, automatic context, uh, summarization. It

also has to have infinite patience or near infinite patience. In this case, this agent loop is simplified to only eight steps maximum. All these considerations, right? Depends on the, on the problem. Yeah. I would start from scratch because that way you can, you get to s- uh, strip 90% of what you don't need. Um, so I would definitely, yeah. I mean, everything is built around the chat completions API, and both Anthropic and OpenAI have created their own standards to fight off against this,

but everything goes around that API. So if you just think of, uh, the way to communicate with a model just as an a- an additional plugin to the, to the harness, it becomes much clearer that the smaller the everything is, the better it is at the end. Does that answer the question? All right, so let's do, uh, this. You guys, uh, start with the code base, um, the... with the codespace. Just set up the notebook, uh, by yourselves, and then we're

gonna go... I'm gonna go through and do a round, and if anyone has any questions, we can chat in a less, uh, aggressive way, because I feel observed, and you probably feel no pressure. So I'm gonna put the pressure on you by stepping off, and if you... Yeah. Uh, let's, uh, keep going. I mean, we don't have a lot of time. Oh- oh my God. Um, I... Time flew. Um, but yeah. Yeah, thank you. Thank you for the clap.

Um, yeah, thank you. Thank you. Uh, so I felt like 15 minutes or 20 minutes went by, but one hour went by. So,

one thing, uh, so I'm gonna leave. Uh, I set this up so that you can create the codespace in the hotel, at home. You can... I will pay for it myself. So you can, you know, use this workshop and, and play with it as, as you wish, and I will pay for it. So don't worry about it. Just, um, if you wanna chat about some of the agent harness concepts, just come to the Oracle booth. We're gonna be here until Wednesday evening, and thank you for coming to the, to the session. Yep.

Okay.