From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Hey, so I'm Anders. Um, I'm a developer advocate for Oracle AI Database, and I'm talking about how we can approach agent memory as a systems problem, how we can take context and turn it into a real memory layer that is going to help your agents work better. Um, when we're talking about agent memory, we're talking about composing memory retrieval, memory formation, and the persistent storage that's backing these memories. Um, and

we've probably all used different agents at this point, agents like Claude Code or Codex, and we've seen them work on a task and reread the same files over and over again, and this, this is because they have no knowledge of what has happened in the past. Context windows are, are not memory. Um, they are the input for the current run and not a persistent experience. If I have some agent session, let's say he's working on customer support tickets. He reads a bunch of documents. He runs a bunch of

queries. He reproduces a failure, finds root cause, and does all this, like, great, valuable work that synthesizes a useful procedure. What happens at the end of the session? It's tossed away, and this is because there's no persistence. Uh, you know, a long, a long later, another, another agent comes on, and he works on a similar task, and he has to repeat all of the same work. So he's, he's searching more documents. He's running more queries, and this is more opportunity for failure, more opportunity for hallucinations,

and, and ultimately, uh, less fidelity. And memory really cannot begin until these useful experiences from our agents are transformed into managed state. Uh, larger context windows are also not the solution. We end up throwing more and more tokens. The session grows longer. Compaction occurs. The useful signal that occurs for our agents is lost in a slew of other irrelevant information. Likewise, scratch pads are workarounds.

So things like markdown files. We start building large hierarchies of files, uh, wikis, prompts. These do help, but this is not governed recall. Uh, files do not scale in large multi-tenant environments. Uh, tools like Grep are extremely useful for finding data, but they also don't scale well, and it is not complex hybrid retrieval. And I feel like if we continue the logical conclusion of trying to build a

memory layer on top of files, we're gonna be building our own database, and I don't-- You could just use a database. Uh, in this presentation, this is unpacking the lessons that we've learned as a database vendor working with customers who are developing agent memory, customers who are solving real problems with agents and needing recall. And where we're seeing modern agent memory systems converge is there's, there's a pattern that's implementing memory

formation, memory recall, evolution, and evaluation. So when we form memories, these are, these are distilled experiences, structured facts. This is not a raw transcript dump. When we're recalling memories, hybrid retrieval is emerging as a dominant method. This is, this is a form of retrieval that fuses different types of search, so vector similarity search, lexical search, entity search over relational objects, graph traversals, and different signals over temporal data, recency,

and feedback. We also need consolidation in these systems, so duplicates can emerge. We need to resolve conflicts. The system needs to evolve over time, otherwise it becomes stale, irrelevant, and useless. Memories need element of temporality. So when was this created? When was it accessed? How long is it valid for? It may only be valid for a certain time window. And this all needs to come into regular feedback and evaluation. If I'm not measuring use, I'm not sending feedback into the memory layer,

it's all gonna fall over. Um, another important piece is the governance controls. So secrets, personal information, that needs to be redacted before it hits the memory layer. Rights need to be scoped, scoped to the agent, scoped to the workflow, the team, the user. This is especially important in multi-tenant environments. We need appropriate admin approvals, review, and ability to suppress stale or poison data. When we walk through the memory loop, we

have an agent session. It does some work, reads documents, run queries, synthesizes useful data. After that session or during that session, a fact is created. We record that fact through redacting, enriching, storing it in the persistence layer. And then when another session comes along with related context, it can retrieve that fact and use it for context card reconstruction. So it's using that fact that came from a prior arc to make the next task easier. That's less work for the

next agent to do. And if that agent uses the fact successfully, we can add feedback scores to that memory. So we're adapting over time. We get related data and new temporality. We access it again, we're-- we did something with it.

Now, memory, it's invoked as a function by the agent runtime. So it needs read, write, and storage layers. Your agent runtime can invoke the memory layer through things like hooks, MCP, skills, or other methods. And the memory layer API is exposing these, the read write, the feedback, administrative APIs, and then talking to the storage platform or databases.

The choice of database here, uh, I, I think multi-model databases are especially useful because- They allow you to not work, work with your rows, columns, vector embeddings, full text search, relational entities, graphs, episodes, and facts all in one place. So your memory can live in a consolidated place with one database query connection. I can do all sorts of hybrid search and return that data back.

Uh, what does the memory look like? Uh, it's gonna be typed, and this ... You may have seen this model at different presentations or some form of it. Um, this loosely comes from human memory, it can be applied to agents. So we have episodic memory. This is specific past events or slices of transcripts that the agents worked on. We have semantic memory. Um, these are specific facts about our workflows, user preferences, and other entities. We have procedural memory. This is, like, how we did a specific task or routine, maybe a build

step. These are reusable rules the agents are gonna wanna pull up again and again if they are repeating similar work. And we have working state. Working state is like short-term memory. It's what are we, what are we current- currently working on, and that, that temporal data. It may only be valid for a certain time window, and we may promote it to a different type of memory. This helps us bucket into different areas, and we can use different search techniques on these types of memory.

Uh, what does the storage scheme look like for memory? Um, so memory are typed objects plus lookup path. So our, our memory object is gonna have some ID, like a primary key. It'll have its text content, JSON tags, document data. It'll have a vector embedding, uh, with a vector created over the content, with a vector index for similarity search. Ideally, we'll also have a text index to facilitate full text keyword search. That combines with vector search. Uh, we can have graph links to different

edges. Memories can be related through an entity graph, so this memory is connected to this memory, which is connected to that memory. This additional context that we can pull back during the search stage. We can also have temporal facts. When is this memory valid to? What was the last time that we used it? And then also, super importantly, audit events. Who, who created this memory? What did they use it for? Uh, the log of, of, of, of how it's been used and associated metadata. Um, this should all be part of your storage schema

when we're working with memories. And when we create memories, we want to intentionally form them. Good memories are very intentional, so we need to decide the life cycle, the scope, the different relationships, and associated metadata. When an agent runtime decides to create a memory, we should first start with redaction. Uh, redaction's very important as a first step because before I do embeddings, before I do enrichment, I want to strip out PII. I don't want secrets or, or, uh, irrelevant, weird information slipping into the memory layer. That's a recipe for

disaster. So we do redaction first. Then we can enrich the memory. What is the mem- w- what is the memory related to? What is the metadata of the memory? We can apply the scope to it. Uh, what is the agent that worked on this memory? What's the team? What's the organization? This helps bucket into specific areas, so I'm not getting memories in other sessions that are associated with other agents or workflows, which adds confusion. We do the embedding, so we create the vector of the content, and then we can form what's called the episode. These are the slices of the

transcript that created the memory. This is good for provenance and additional context. Um, any organization-specific policies here can also be applied, uh, whatever you need for your specific use case, and then we can save down to the storage layer. Um, when we create memory, the individual memories can be quite compact, so we take the transcript and we distill it down to that content and metadata and embedding and, and, and related stuff, but we can also ... We create these episodes. So we take the agent transcripts,

we filter them, find candidates for these episodes, save the memory record, and link it back. You can do this with, uh, relations, just straight-up database relationships. You can do it with graphs. I particularly like graphs because they allow a, a broader search here.

Uh, when we're talking about memory recall, that is, how do we remember something, this is a retrieval problem, so a query problem on the storage layer. Um, and with, with retrieval or recall, hybrid search is emerging as the dominant method of context reconstruction, and this, this form of retro- retrieval can fuse different search methods. So I can take vector similarity search, I can take a full text keyword search, I can take relational entities, I can do

graph traversal, I can take other feedback signals like importance, uh, uh, usefulness of the memory to yield more comprehensive revo- results than one specific query branch alone. Like, if I was only doing similarity search, I only get the most similar results according to the embedding model that I'm using. Um, and in general, this has not been enough, so we have all these different query branches, um, different types of memory. You can decide which ones you wanna do, things like facts. You might not want similarity. You might just

wanna do, like, an exact ID match on a fact. Um, we can take these results, we can fuse them, apply thresholds, do re-ranking, and then we get a context card with scored evidence. So we can say, this memory, it rate- it rated very well on vector search and a little bit lower on lexical search. Um, that's the provenance of the memory, where it came from, how we got it. That's important, too.

Um, as we, as we walk through hybrid search, uh, vector distance, useful for, for semantically similar, things like contains operator, very good for lexical search, uh, graph table, temporal data, all this stuff. We take these branches, we apply weights to them. We can weight differently as needed. This is something that can be tuned in your, your hybrid search algorithm. The m- the results are merged to an overall score from the different query branches, and then we can take the top K of the overall results to say, "Hey, these are

the top three best results from our hybrid search algorithm." And that's loosely how you could implement hybrid search in a multi-model da- uh, database, a database that is capable of providing all of these different search methods from one pane of glass here.

Now memories, it's not just a repository where we push data in and we pull data out. Memories need to evolve over time to be truly useful. We also don't want to just shove all our data in every time, otherwise we get tons of memories. Um, so and this is where we can kind of close the memory loop with consolidation and error handling. Uh, stale memory can occur when, say, my, my project uses NPM at one time, and then we decide to switch to another build system. But what

happens to all the memories that were related to NPM? If an agent recalls those, it's, it's gonna be a little tricky, and it may try to do the wrong thing. Uh, so the system, it needs to expect stale facts over recall quiet drift. Bad scope can occur where, hey, we're not appropriately, ap- appropriately scoping memories in the right context. This is like if you log into your ChatGPT account and you get someone else's message history. Um, if you're not applying scope properly, you can have some really bad data leaks. So this needs to

happen on write and on read. Uh, poison memory is when a untrusted source plants a durable instruction in your memory layer. This is kind of the worst case scenario, like a, a prompt injection happened, you didn't have the appropriate guardrails, you didn't have the appropriate review, redaction, controls on memory and, and some really nasty stuff got in there and all your data got exfiltrated. So, uh, be very sure to avoid these kind of things through the proper guardrails and administrative actions. Uh, a memory

conflict is when I have two different memories. One says, "Hey, we do the task this way," and the other says, "Well, actually, we do it this way." Because they're both about the same task, they're gonna rank very highly on retrieval, so you'll probably get these same conflicts in your context card. And when the agent consume that, it's like, "Well, you know, you told me the restaurant's to the right, and it's actually to the left. I don't know what to do." Um, so detection review here is very important for conflict resolution. Uh, scoping as well, sometimes there

are different ways to do the same task. Um, over-recall and irrelevance happen as your memory system grows. Um, if we're just shoving all of our data in there and not really thinking about what we're writing, um, we can get into the situation of over-recall, where we have so many different weak matches, they're crowding out the context card. This is the same problem with the very large context window, where we try and smoosh in as much data as possible. It's just confusing. Irrelevance, uh, semantically close, memories that

rank highly on hybrid search, but it's not actually about the task. It just, it rated highly. So these, these are problems of, of management and your search algorithm tuning. Data leakage is when we get weird, unexpected data that enters the memory system. This is a less serious version of poison memory. Um, proper enrichment and distillation of the memory records on the right path is the way to solve leakage. Uh, over time also, we get silent drift. This is where the memory system grows. Maybe it works

freaking awesome at, like, a thousand records, but then when I have fifty thousand or a million or more, it's-- I'm getting all these weird results and it's not really working. So it's very, very important to apply the proper countermeasures, the redaction before write, the scope checks, review, life cycle, and most importantly, feedback. Feedback is what closes the loop for memory. Memory quality needs to be measurable. We need to have correct retrieval. We need to account for stale or harmful memories, and ultimately, the downstream usefulness of the

system, it needs to feed back to these ranking and life cycle decisions. So if an agent retrieves a candidate memory, it'll use it, it'll reject it. Maybe if it uses it, everything goes haywire and it crashes. Like, that feedback needs to come back, and if, if the memory was harmful, the feedback should go down. If it was useful, maybe we promote it. So it's not just a repository, it needs to store-- your memory layer needs to store whether memories were helpful, stale, irrelevant, harmful, et cetera.

Um, and we really, we want to design for measurable recall with governed memory formation built on a mul- multi-model storage platform. So without memory, these agents are more likely to restart the same tasks, they're gonna recreate over and over again, they're gonna read the same files, they're gonna run the same queries, they're gonna waste tons of time, tons of tokens, um, be much slower, and much more, much more failure-prone. They're gonna repeat the same mistakes. If it doesn't know about prior art, it's gonna do the same thing over again, and if that

thing was wrong, it'll be wrong again and again. And with governed memory, the agents can resume where they left off, they can reuse what came before them, and ultimately reduce repeated failures. This is, this is the true goal of recording, retrieving, governing, evolving. Um, we want to improve the success rate. We wanna do it faster, we wanna do it cheaper, we wanna do it better.

Um, if you are hands-on, um, I recommend checking out our Python SDK for Agent Memory. This code will take you to a GitHub link with a Python notebook that walks through how to do this using our SDK. The SDK is buil- built on database primitives. Um, everything that we talk about here lives in the database. Um, you-- We also have our developer hub. If you're an AI engineer, you're a builder, you're curious, like, how does a modern multi-model database fit into AI

development, this is the place to go to check out our samples, um, see what's possible, and maybe even inspire you to build something cool, uh, with a very awesome database. Uh, again, I'm Anders, um, Anders Swanson. If you want to chat or connect with me, I'll be over at the Oracle booth. Um, and thank you so much. I hope this was useful to you guys and, uh, helpful as you're, you're building awesome things. Thank you so much.