Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, Oracle

Language models usually arrive with fixed weights, yet the applications built around them need to improve and produce dependable results. Ignacio Martinez, an Oracle developer advocate, locates that engineering work in the harness: the memory, tools, and environmental inputs surrounding the reasoning model. His workshop begins with a registration site he created the preceding Saturday and a GitHub Codespaces environment. The architecture underneath that improvisation is deliberate. 4:02

Looking ahead to the agent definition Martinez gives after the stack, connectivity and application shapes: An agent, in Martinez’s formulation, is a model plus a harness. The model supplies reasoning; files or databases supply memory; tools enable actions; inputs let the system perceive its environment. You rent the reasoning service and accept what its provider offers. Memory, tools, and perception give you room to customize how the system works.

The five-layer agent stack separates the application users interact with, its data, the reasoning model, infrastructure that orchestrates model serving, and underlying compute. Martinez sees much of this platform becoming commoditized, leaving data as a particularly useful place for builders to exercise judgment. Data includes memory, knowledge, encoding, search, and retrieval—not merely records fetched at the end of a request.

Connectivity makes that information actionable. A gateway and Model Context Protocol (MCP) layer expose functions that connect the model to external applications and tools. Outlook is the example: a model without access cannot operate the application; exposing suitable functions lets the surrounding system communicate with it. Memory, retrieval, semantics, context, tools, and skills build on this connection.

Different applications give the model different amounts of initiative:

• Chatbots: Respond when a user asks a question.
• Retrieval-augmented generation (RAG) applications: Process and retrieve supporting information while remaining largely request-driven.
• LLM-driven workflows: Automate a sequence of work.
• Agents: Add autonomy to choose how work proceeds.

Coding systems such as Claude Code and Codex combine workflow automation with agent autonomy. Automation contributes reliability; autonomy contributes flexibility.

Harness engineering aims for repeatable outcomes around a nondeterministic component: the same input can produce different model outputs. The harness organizes execution without assuming that reasoning has become deterministic. A common interface, such as an OpenAI-compatible protocol or an Anthropic API specification, also allows the reasoning service to be replaced while retaining the surrounding system.

The workshop introduces a seven-layer harness, then develops its responsibilities through storage, memory engineering, semantics, agent loops, and context engineering. Storage comes first because later operations depend on where information physically lives. Encoding and retrieval make it findable; semantics explains local vocabulary; context engineering keeps the current task salient. These changes can affect behavior while the model’s weights remain fixed. 13:48

Files are easy to create, read, and append to, and they accommodate unstructured material. Their familiar operating-system interface makes them convenient working memory. A database need not replace that interface everywhere. Martinez prefers using files and databases together, assigning each the work it handles well.

Parallel agents expose a weakness of ordinary shared files. Several agents may try to change the same file, and the file interface alone does not coordinate those changes transactionally. Asked how coding agents avoid trampling one another’s work, the room answers: worktrees. Each agent changes a separate working copy, then integrates its implementation into the main branch. Isolation moves the collision to a later merge; it does not turn concurrent updates to one shared file into a transaction. 18:35

Databases bring established data-management capabilities:

• Transactions: ACID operations provide atomicity, consistency, isolation, and durability.
• Availability: Replication can keep data available across multiple locations.
• Retrieval: Vector and hybrid search support finding information beyond ordinary text matching.
• Recovery: Backup facilities address failures that an unprotected local file collection does not handle by itself.

These are the problems behind Martinez’s observation that databases solved much of this decades ago and people have forgotten.

A proposed exercise puts three agents on the same counter file to compare update behavior; its outcome is not shown in this recording. Oracle DBFS, the Database File System, offers the architectural alternative: retain a file-facing interface while storing files inside a database, gaining transaction and management capabilities. A simpler hybrid allocation keeps short-lived working material in files and promotes lasting information, such as user preferences, into structured database storage.

Once storage exists, the harness needs to encode, search, and retrieve information. LangChain Oracle DB supplies an integration for vector-store operations. In-database embeddings place the embedding model inside the database, avoiding a third-party service call during encoding. For enterprise teams, that changes where data must travel and helps control its handling. 21:34

The retrieval pipeline has two matching stages. A bi-encoder embeds document chunks and the user’s query into vectors that can be compared in a vector store. This selects candidate material. A cross-encoder then considers the question together with retrieved results and reranks the candidates. The first stage finds plausible matches; the second evaluates those matches alongside the question.

How does a question meet the documents that might answer it? The diagram separates document preparation from the query path, then shows where they join. Reranking follows vector retrieval: it evaluates selected candidates rather than replacing the initial search.

This pipeline multiplies the kinds of information an application manages. Preparation involves tokenization, normalization, embedding generation, and redaction of personally identifiable information. Storage then includes source text, JSON metadata, and dense vectors. Splitting these across databases creates synchronization and maintenance work: the application must keep the representations aligned as data changes.

Oracle’s converged-database proposal addresses that coordination cost by placing relational, JSON, spatial, graph, and vector-oriented work in one database engine and query environment. Consolidation can simplify the development stack and reduce the number of systems to secure and maintain. Martinez’s security simplification does not establish that the resulting application has only one possible attack path.

Agent memory comprises mechanisms that retain, recall, reuse, and refine information. Its purpose is practical: a problem that took three hours should become easier when it appears again. That requires separating information useful during the current task from experience worth carrying into future work. 25:53

The memory categories answer different needs:

• Short-term memory: A coding agent’s current to-do list helps it finish today’s task and need not become permanent history.
• Episodic memory: Previous conversations preserve experiences that can inform improved workflows.
• Procedural memory: A successful process becomes reusable instructions—for example, the workflow behind a front-end design the user liked.
• Shared memory: Parent agents, subagents, or collaborating agents exchange information needed to solve a problem together.

A larger context window does not perform all these jobs. Context is working memory, and filling it with history can make the current task harder to follow. Martinez calls this context rot. His analogy starts with thirty minutes of attentive conversation and stretches into eight hours: eventually the listener wants him to stop talking. Keep useful information close and unnecessary history out.

The attention-matrix explanation gives a concrete scaling relationship. Each token relates to other tokens, so a full token-to-token matrix grows in both rows and columns as context grows. Doubling the token count produces four times as many matrix entries. That explains the quadratic relationship described here; it does not establish a universal rate of answer-quality degradation or a safe context threshold.

Memory engineering includes deciding when to compact, how to summarize, what to preserve, what to extract, and how many tokens to spend. The Oracle Agent Memory Package, or OAMP, is presented as a managed way to assemble a context card through a single Python call. It packages work the engineer would otherwise have to coordinate. 31:29

The context card separates information by its role:

• Topics: Orient the model toward the conversation’s subject.
• Summary: Compact the thread and preserve current intent.
• Relevant information: Bring forward associated facts, preferences, and memories.
• Episodic memories: Track the unanswered question still in progress.
• Recent messages: Preserve immediate conversational context.

An audience question clarifies where this structure operates: the harness uses it to shape what goes into the model. The card organizes inputs around the reasoning core; it does not update model weights.

Memory preserves information; semantics supplies meaning people often leave unstated. Martinez borrows Umwelt from Jakob von Uexküll to describe an organism’s perceptual lens: it experiences the world through what its senses make available. An agent’s corresponding lens consists of its training and the context it receives. 34:33

Colleagues can discuss work without explaining every internal term because they share institutional knowledge. A newcomer needs those missing explanations. The semantic layer gives an agent that background—how data is modeled, how queries are executed, and what metadata means. Otherwise, requests pass through a lens missing the organization’s assumptions.

The agent loop makes this knowledge actionable. Its minimal cycle is observe, reason, and act, repeated as new information arrives. The loop drives the model and gives it autonomy to continue work. Failure resistance matters because an unsuccessful action should become information the system can respond to rather than automatically ending the task.

Context assembly belongs inside that cycle. The toolbox and skill-box patterns keep tools and skills in storage, retrieving them when the current task needs them. At each iteration, the harness can remove material that is no longer relevant. A capability can remain available without its entire description continuously occupying the context window. 37:33

Continual improvement can happen in model weights, representations used by embedding and reranking systems, or context supplied to the model. The workshop focuses on context and token space because Martinez considers it an accessible, inexpensive way to change behavior while keeping the reasoning core frozen.

Skill promotion develops the earlier front-end example. A session produces a design the user likes, and the conversation contains the process that reached it. Store that experience, distill the successful workflow into a revised skill.md, retire the old version, and retrieve the new skill for later tasks. A reusable instruction artifact now carries the procedure that previously lived in a one-off conversation. 39:36

Preferences become part of the procedure: a library whose look and feel the user likes, a database engine they found easier to use, or a preferred tone. The workflow becomes more personal as these choices accumulate. The mechanism preserves and revises instructions; no criterion for judging the distilled skill’s improvement is established here, so its benefit depends on the quality of promotion.

The hands-on environment pairs a student notebook with an AppBook for interacting with individual harness components. Both run in GitHub Codespaces, with model requests served through OCI Generative AI Service. Martinez asks participants to avoid eight- or sixteen-core machines because he is paying for their Codespaces himself. Wi-Fi trouble and a Codespace disconnected through inactivity interrupt the transition to the demonstration.

The notebook grows the system in layers. Its first exercise sends a question through the OpenAI Completions API and receives a response, with no harness implemented. Later exercises add search, retrieval, encoding, and other components. There are nineteen to-dos. Starting with a bare reasoning call gives participants a baseline against which to understand what each surrounding layer contributes. 48:10

The complete Total Recall application is exposed through a public Codespaces port for browser access. Its concrete request asks for total revenue by product category. Mission control reveals the tools selected for context, the schema available to the agent, and traces of loaded skills, data sources, and tool calls. The question becomes an inspectable sequence of work against database information. 50:07

During the sequence, the trace reports an error. The agent loop continues trying, and the application subsequently presents a database-derived result. Martinez describes the run as taking sixteen steps. The revenue values and particular error and repair are not supplied, so the observable change is from a request, through selected context and a recoverable failure, to a result—not a verified financial calculation.

What lets the revenue request survive an error? The diagram connects selected context to actions and routes the reported failure back into continued work. That return path explains how one unsuccessful action can become part of a longer attempt. The application also exposes token usage and lets users interact separately with context-card creation.

What happens when an organization has thousands of tools? The toolbox pattern searches stored tool representations instead of placing every description in context. Martinez proposes an HNSW index—hierarchical navigable small world—as the graph-based search structure behind retrieval. The model receives a relevant subset while the larger inventory remains in storage. 51:58

The database-backed index keeps the application’s query interface simple as the inventory grows. Martinez describes retrieval overhead as negligible; that is a design expectation here, rather than a measured latency result or evidence that search cost remains constant between one thousand and ten thousand tools.

A harder question concerns tools from different business units with overlapping descriptions and different access to confidential data. The retrieval proposal is to use an LLM to enrich descriptions or docstrings, making the tools easier to distinguish in vector search. This addresses semantic separability. Permission enforcement is not specified in the answer, so better descriptions alone do not resolve which confidential data a caller may access.

Failure resistance creates another problem: an autonomous loop can keep spending money without approaching its goal. An audience question brings the discussion back to stopping. Martinez uses a model-dependent cutoff expressed as a maximum number of tool calls before giving up. For the model used in this workshop, he reports finding eight to twelve calls a useful range. This is his tuning observation for that workshop model, not a recommendation for a verified model release or for other models. 55:45

He calls the control a hysteresis variable: the amount of patience the harness extends to the model. Too little patience abandons recoverable work; too much prolongs unproductive generations. He contrasts a result reached in two steps with the earlier sixteen-step run. Those steps and the proposed tool-call limit are not clearly the same counter. Eight to twelve is a tuning observation, and the cutoff is a discussed harness control rather than an established notebook implementation.

Model routing supplies another control. Difficult problems can go to a frontier language model, while easier ones can use a smaller open-weight model. The separation between reasoning and harness makes this possible: the surrounding system chooses both the reasoning resource a task receives and the patience allowed for continued attempts.

Martinez’s forecast is a mixture of small experts: specialized models, potentially around one hundred million parameters, coordinated by an orchestrator that routes queries to the appropriate model. He expects that arrangement to make a harness more token-efficient. It is a proposed direction, rather than a demonstrated replacement for the workshop’s general-purpose model. 57:17

The recording ends by returning participants to implementation. The notebook uses a Python 3.12 kernel, and its docs folder provides explanations for individual exercises alongside available solutions. The progression begins with one model call and adds harness components around it. Staff move into the room to help participants work through those layers, with internet access still causing trouble as the hands-on session begins.