AI Engineer World's Fair 2026

From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Loading slides…

Read the talk

From Context to Memory: Your Agents Need a Real Memory Layer

Anders Swanson explains how to turn useful agent experience into managed state: form compact memories, retrieve them through several search methods, and use governance and feedback to keep yesterday’s knowledge useful tomorrow.

From a talk by Anders Swanson

At a glance

Ideas worth remembering

  • Useful experience becomes memory when it survives the session as managed state that another session can retrieve and reuse.

  • Redact before enrichment and embedding, preserve provenance through linked episodes, and enforce scope on both writes and reads.

  • Hybrid recall combines semantic, lexical, relational and graph lookup with time and feedback signals, then selects a compact context card with scored evidence.

  • A memory’s relevance does not establish its validity or usefulness. Lifecycle management, conflict review and downstream feedback keep persistent knowledge from repeating persistent mistakes.

The next support ticket starts from zero

A support agent reads documents, runs queries, reproduces a failure and finds its root cause. The investigation produces something more valuable than an answer: a procedure another agent could reuse. But if that procedure disappears when the session ends, the next similar ticket starts the investigation again. Anders Swanson, a developer advocate for Oracle AI Database, opens with this distinction between information available during a run and experience that survives it.

“Context windows are not memory.” A context window supplies the current run’s input; persistence lets a later run benefit from earlier work. In the support example, losing the procedure means searching documents and running queries again. Each repeated step consumes time and creates another opportunity for an incorrect inference. The useful transition happens when the investigation’s result becomes managed state that a future session can retrieve.

A larger context window does not make that transition happen. More tokens can keep more material in the current session, but long sessions still accumulate irrelevant information and undergo compaction. Markdown scratch pads, wikis and prompt files do preserve some knowledge. Their limitation is the work required to make recall selective, scoped and manageable across many users. Grep can find text; it does not supply the combination of search methods and governance this memory layer needs.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 94 secondsOpen full source frame
Without persistence, the second agent session repeats the investigation.

Swanson’s database-vendor perspective gives the argument its blunt punch line: keep adding retrieval, access controls and management to a file hierarchy, and eventually you are building a database. “You could just use a database.” The rest of the talk explains what that database-backed system must do beyond keeping text on disk.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Preserve the useful result, then learn from its reuse

The proposed pattern has four connected jobs:

  • Formation: distill experience into structured facts rather than save an undifferentiated transcript dump.
  • Recall: combine search methods to recover information relevant to the new task.
  • Evolution: consolidate duplicates, resolve conflicts and track when a memory remains valid.
  • Evaluation: measure how memories are used and feed those outcomes back into the system.

Governance applies throughout this cycle. Secrets and personal information must be redacted before entering the memory layer. Access must be scoped to the relevant agent, workflow, team or user, especially when multiple tenants share infrastructure. Review, administrative approvals and suppression provide ways to keep stale or poisoned records from continuing to influence agents.

Return to the support investigation. During or after the session, the agent creates a fact from the useful work. The system redacts it, enriches it and stores it. A later session with related context retrieves that fact into a context card—the selected information supplied to the agent for its next task. The observable change is that the later agent can start with the earlier result rather than repeat all the discovery. If reuse succeeds, feedback updates the memory’s score and records its renewed use.

Where does one session’s work become another session’s starting point? The cycle below makes the handoff visible. Persistence carries the fact across sessions; retrieval selects it for the new task; feedback changes how the stored fact should be treated next time. Saving the fact opens the loop, but recording the result of reuse closes it.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 282 secondsOpen full source frame
A memory loop connects one agent session to the next.

A support investigation becomes reusable experience

Investigate a support ticketRead documents, run queries, reproduce the failure and find its root cause.
Form and store a memoryCreate a useful fact, redact it, enrich it and persist it.
Reconstruct later contextRetrieve the fact for a related session so the next agent can reuse prior work.
Record the outcomeUpdate feedback scores and information about the memory’s use.
The later session receives a selected fact from prior work, then sends its outcome back to the memory layer.
Read the diagram as text
  • Investigate a support ticket. Read documents, run queries, reproduce the failure and find its root cause.
  • Form and store a memory. Create a useful fact, redact it, enrich it and persist it.
  • Reconstruct later context. Retrieve the fact for a related session so the next agent can reuse prior work.
  • Record the outcome. Update feedback scores and information about the memory’s use.
  • Investigate a support ticket → Form and store a memory: Distill useful work.
  • Form and store a memory → Reconstruct later context: Retrieve for related context.
  • Reconstruct later context → Record the outcome: Observe successful use.
  • Record the outcome → Form and store a memory: Update the stored memory.
3:133:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:13 · section reference included

Give the runtime an API and memories a type

Memory is a function the agent runtime invokes. Hooks, MCP, skills or other integration methods connect the runtime to an API that exposes reads, writes, feedback and administrative operations. That API then talks to persistent storage. These are separate responsibilities: the runtime decides when memory is useful, the memory layer manages its operations, and the database stores and queries the records.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 345 secondsOpen full source frame
The runtime connects to a memory API and storage layer.

A multi-model database offers a practical consolidation: rows, columns, embeddings, full-text search, relational entities and graphs can live behind one database connection. This lets different retrieval methods query the same collection of episodes and facts. Swanson favors that arrangement because it keeps the ingredients of hybrid search together; it is an architectural preference presented here, rather than a comparison of measured outcomes across database designs.

Typing the records helps determine what to store and how to retrieve it:

  • Episodic memory: a specific past event or a slice of an agent transcript.
  • Semantic memory: facts about workflows, user preferences or other entities.
  • Procedural memory: reusable instructions for a task or routine, such as a build step.
  • Working state: what the agent is currently doing, potentially valid only for a short period and later promotable to another memory type.

These categories loosely borrow from human memory, but their purpose here is operational. A reusable procedure and temporary working state should not automatically share the same lifetime or lookup strategy. Separating them creates room to choose a search method appropriate to the information the next task needs.

5:305:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:30 · section reference included

Build lookup paths—and redact before creating them

The storage model is a typed object plus ways to find and interpret it. A memory has an ID, text content, JSON tags and document data. An embedding and vector index support similarity search; a text index supports keyword search. Graph links connect related memories or entities so retrieval can follow relationships beyond a single matching record.

Time and audit history belong in the schema too. When is the memory valid? When was it last used? Who created it, and what was it used for? These fields describe whether the record is still appropriate and preserve its history. A content field alone cannot answer those questions, even if its text looks relevant.

Formation starts by deciding the memory’s lifetime, scope, relationships and metadata. The first processing step is redaction, before enrichment or embedding. That order matters: removing personal information and secrets first keeps them out of the material used to create the memory’s derived representations.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 502 secondsOpen full source frame
Memory formation begins with redaction before enrichment and embedding.

The remaining content can then be enriched with relationships and metadata, scoped to its agent, team or organization, and embedded. Scope reduces the chance that an unrelated workflow receives a confusing memory. Organization-specific policies apply before the record is saved. The point is to make a deliberate write, with enough information to manage the record later.

Compact memories need not lose their supporting context. The system can filter an agent transcript for useful episode candidates, save the distilled record and link it back to the relevant episode. A relational link provides that association; a graph can support broader traversal among related records. This keeps the reusable fact small while preserving a path to the experience that produced it.

7:187:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:18 · section reference included

Several search branches produce one context card

Recall is a query problem: which stored information should enter the current context? Vector similarity provides the results that look most similar under the embedding model. That can help recover related experience, but it does not cover every lookup. A fact about a known entity may be better retrieved by an exact ID match.

Hybrid search combines complementary signals:

  • Vector similarity: find content with similar meaning.
  • Lexical search: find matching words through full-text search or a contains operation.
  • Relational lookup: find facts associated with known entities or IDs.
  • Graph traversal: follow relationships to connected memories and additional context.
  • Temporal and feedback signals: consider recency, importance and prior usefulness when choosing results.

The query branches produce candidates that the system fuses, filters with thresholds and reranks. Branch weights can be tuned, and their contributions can be merged into an overall score. A top-K selection then supplies the context card; Swanson uses the top three results as an illustration. The card can retain scored evidence, including that a memory ranked highly on vector search but lower on lexical search, so the agent receives information about how the result was found.

How can semantically related experience and an exact fact both reach the same agent? The flow below shows independent lookup branches converging on a shared selection step. Their contributions remain distinguishable even though the agent receives one context card. The recording sketches weighting and fusion without specifying a score-normalization method or a particular ranking formula, so those choices remain implementation work.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 691 secondsOpen full source frame
Vector, text, entity, temporal and graph signals feed candidate fusion.

Hybrid recall combines lookup methods before selecting context

Vector searchLookup branches — Retrieve semantically similar content.
Lexical searchLookup branches — Retrieve keyword matches.
Entities and graph linksLookup branches — Look up exact facts and traverse related records.
Time and feedbackRanking signals — Supply recency, importance and usefulness signals.
Fuse and rankWeight contributions, merge results, apply thresholds and rerank.
Context cardReturn top-K memories with scored evidence about retrieval.
Different branches find different kinds of evidence; fusion and ranking determine which memories reach the agent.
Read the diagram as text
  • Vector search. Lookup branches — Retrieve semantically similar content.
  • Lexical search. Lookup branches — Retrieve keyword matches.
  • Entities and graph links. Lookup branches — Look up exact facts and traverse related records.
  • Time and feedback. Ranking signals — Supply recency, importance and usefulness signals.
  • Fuse and rank. Weight contributions, merge results, apply thresholds and rerank.
  • Context card. Return top-K memories with scored evidence about retrieval.
  • Vector search → Fuse and rank: Similarity candidates.
  • Lexical search → Fuse and rank: Keyword candidates.
  • Entities and graph links → Fuse and rank: Facts and connected context.
  • Time and feedback → Fuse and rank: Additional ranking signals.
  • Fuse and rank → Context card: Select top-K results.
10:0410:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:04 · section reference included

Relevant memories can still be wrong, unsafe or overwhelming

Persistence creates a new problem: yesterday’s useful instruction can become today’s mistake. In the talk’s example, a project once uses NPM and later switches to another build system. The old build memories remain searchable. If recall returns them without accounting for the change, the agent may follow an obsolete procedure. Evolution therefore includes consolidation, lifecycle management and ways to suppress stale facts.

Other failures require different controls:

  • Bad scope: a memory reaches the wrong user or workflow. The analogy is opening your account and seeing someone else’s message history. Scope checks must apply on both write and read.
  • Poisoned memory: an untrusted source plants a persistent instruction, allowing a prompt injection to survive into later sessions. Review, guardrails and administrative controls must address what is allowed into memory and what remains available.
  • Conflicting memory: two records describe incompatible ways to perform the same task. Both can rank highly because both are relevant. Detection and review must resolve the conflict; scope may explain cases where different procedures are valid in different settings.
Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 848 secondsOpen full source frame
The slide lists memory failure cases, including conflicting memories.

More memories can also recreate the oversized-context problem. Over-recall fills the context card with weak matches, crowding out useful information. Irrelevance is subtler: a memory scores highly because its meaning resembles the query, yet it does not help with the actual task. Both require attention to what gets written and how retrieval is tuned.

The talk also calls out unexpected data entering the memory system, which careful distillation and enrichment should prevent. As the collection grows, retrieval can drift without an obvious failure event. The contrast between a thousand records and fifty thousand or a million is an illustrative warning about scale, not a measured capacity limit: a search configuration that works on a small collection may return increasingly strange results on a larger one.

12:1312:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:13 · section reference included

Measure what happened after retrieval

Feedback is what makes memory adaptive. Retrieving a candidate does not prove that it helped. The agent might use it, reject it or follow it into a failure. The memory layer needs to record whether the result was helpful, stale, irrelevant or harmful, then feed that information into ranking and lifecycle decisions. Harmful use should lower its standing; useful experience may justify promotion.

Selected presentation frame from From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle at 942 secondsOpen full source frame
Feedback from memory use informs later ranking and lifecycle decisions.

This adds a second question to evaluation. Did retrieval find an appropriate memory, and did that memory improve the downstream task? A highly ranked record can fail the second test. Tracking both connects search quality to the work the agent actually performs, rather than treating retrieval scores as the final measure of success.

The intended payoff is practical: agents resume work, reuse prior experience and repeat fewer mistakes. In the support example, the retained procedure saves a later agent from rediscovering what the earlier investigation learned. Faster execution, lower token use and better success rates are the goals of the design; the recording does not quantify those gains for this system.

For implementation, Swanson points to Oracle’s Python SDK for Agent Memory and a Python notebook walkthrough, describing the SDK as built on database primitives. He also directs builders to the Oracle AI Developer Hub for samples showing how a multi-model database fits into AI development. The closing invitation follows the architecture: implement formation and recall on persistent storage, then make governance, evolution and feedback part of the same working system.

15:1215:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:12 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…