Everyone Has the Same AI Models. Here's What Sets Agents Apart - Richmond Alake, Oracle

A year earlier, Richmond Alake had asked an audience about agent memory and found it largely unfamiliar. This time, hands went up across the room. His shorthand for what a memory system does is the four Rs: retain, reuse, recall and refine information. Short-term and long-term describe how long something stays useful. A different question is what the agent needs it for. Semantic memory holds facts, episodic memory records what happened, procedural memory captures how to do something, and working memory keeps information at hand for the current task.

Fewer hands went up for control plane. This is the management system that establishes an agent’s identity, grants tool permissions, applies policies, orchestrates work and keeps an audit trail. A software factory uses repeatable workflows to produce software, often with several agents executing in parallel. Alake traces the factory idea back to work at General Electric in 1968. Both concepts answer practical questions about getting work done reliably. Neither tells an agent what a term means inside the business that employs it.

By the time Alake got to intelligence plane, he was hoping for a roomful of blank looks. Three people raised their hands. To explain what he meant by the term, he turned to a much older problem of interpretation.

In Alake’s account of World War II codebreaking, teams in Washington and Pearl Harbor had fragments of intercepted Japanese naval communications. One designation mattered enormously: AF. At Station HYPO in Pearl Harbor, codebreakers working in a basement had kept paper records of intercepted messages for months. That accumulated record let them consider fragments together and form a hypothesis about the place they referred to. AF meant Midway.

Washington had centralized control and, in the telling, access to the same information. It reached a different interpretation. The American Navy acted on Station HYPO’s reading. The part of the analogy that carries over to agents is the distance between possessing a message and knowing what its references mean. Teams can buy access to the same model and still differ in the experience they preserve and the meanings they can recover from it.

Alake uses intelligence plane for a set of responsibilities that cut across the stack. What should the agent know? What should it remember? Which data may it see on this user’s behalf? How can an earlier run improve a later one? Teams that began by inspecting prompts, then inspected the context around those prompts, and eventually built orchestration and controls already have places to implement the answers.

• Gateway and connectivity. Carry the end user’s identity through to the data. Knowing which agent connected is only part of deciding which records it may read.
• Orchestration. Assemble the context for the next model call, including relevant memory. Choosing what the model sees is part of executing the task.
• Database. Store memories alongside an ontology that describes the domain’s entities and a semantic layer that defines its terms. These give the agent a way to interpret the records it retrieves.
• Infrastructure. Provide inference for the models in use and a way to fine-tune and deploy updated models as the system accumulates experience.

There is a precedent in networking, where proposals for a knowledge plane extended the systems used to manage and configure routes. Applied to the shorthand agent = model + harness, governance and security constrain what the running agent can do. Knowledge, memory and learning influence which decisions it makes within those constraints.

An agent runtime needs knowledge and memory while it executes. In Alake’s reference architecture, a memory manager sits over a data platform. It supplies memory to the runtime and accepts traces from the runtime about its interactions. The runtime, manager and storage can come from different providers.

What comes back to the runtime after it records a trace? On a later request, it can get remembered information to include in context. There is also a longer path. Captured traces can supply examples for fine-tuning, and the resulting model can go back into the runtime. In the first case the model receives different input. In the second, training changes the model that will process future inputs.

Small decision models make the second path practical for a narrow job. A retrieval pipeline has decisions inside it long before the main language model writes an answer. One example is deciding where to file a new memory. A team can train a model for that one choice and put it into service independently of the model that talks to the user.

The first demo compares a governed analyst with a naive analyst in a chat interface. Both receive the same questions. The governed version has the ontology, semantic definitions, guard controls and prompt caching. Alake reports 99% correct answers for that version and token use of 32,000 versus 200,000 for the naive one. Those figures describe the demo configuration as a whole. The comparison leaves the question count, scoring method and each component’s share of the savings unspecified.

What changes in the answers when the agent has the firm’s meanings available?

• Counterparty tests the ontology. Keyword search finds some entities, enough for the naive analyst to produce an answer. The governed analyst can also consult a model of what those entities represent in the domain. On the counterparty question, that extra information yields the answer Alake identifies as correct.
• Concentration tests the semantic layer. Asked what concentration means, the naive analyst gives the wrong interpretation. Providing the domain definition through the semantic layer lets the governed analyst answer the same question correctly.

The counterparty example is easy to miss because retrieval appears to have worked. A search hit establishes that the records contain a matching term. An analyst still has to decide what those records mean for the question. The concentration example makes that distinction especially clear. A finance team that already knows the intended definition can write it down once and make it available whenever the term comes up.

Treating memory as data gives a team concrete work to do. Write down entities and definitions, put entitlements in the data layer, and capture traces as the agent runs. Oracle Agent Memory packages some of that work in Python. It generates context cards to bring remembered information into a request. Saving memory traces serves a separate purpose: it gives the team material for training later. The Oracle Agent Memory report develops the distinction between managing memory and storing it.

Suppose an analyst asks the agent to look something up. Whose permissions determine which rows come back? With Oracle’s Deep Data Security, the identity provider supplies the analyst’s entitlements. The agent inherits those entitlements when acting on the analyst’s behalf. The application carries that context into the data layer, where the database restricts access to the rows that analyst may see. Permission to use a database tool and permission to read a particular record are two distinct decisions.

Where inference runs is a separate deployment choice. A financial institution that needs on-premises inference can use the Private AI Services Container to host embedding models, language model inference and rerankers, including on air-gapped infrastructure. Oracle Cloud offers a hosted route. The row-access rule answers what an analyst can retrieve. The hosting decision answers where models process it.

In Alake’s MemoRizz demo, the memory system has three model jobs. A decision model chooses where an item belongs. An embedding model represents items for retrieval, and a reranker orders retrieved candidates by relevance. The demo follows the first job. When output arrives from the context layer or the language model, should the system file it as part of a conversation or as part of a workflow?

What can teach a router that distinction? Alake took material from his own conversations with the agent and classified it by memory type. Those pairs supplied the training input: an example of incoming content together with the kind of memory it belonged to. He fine-tuned a Qwen model on the examples. The resulting decision model takes over a choice a developer might otherwise encode in routing rules.

The training examples and a newly arriving item play different roles. The labels shape the router during training. At runtime, the router predicts a type for the new item, and that choice determines its destination. This keeps a frequent decision small enough to make cheaply. Embedding and reranking models are other candidates for tuning on retained traces, with potential gains in retrieval quality and, for some workloads, latency.

Oracle also exposes its memory system through Private Agent Factory, a drag-and-drop agent builder with a control plane built on Oracle AI Database. A team can assemble an agent there and use the same underlying memory capabilities. The financial analyst still depends on somebody supplying the business definitions. Even a small accuracy difference can matter if the answer informs how a firm handles a potentially fraudulent transaction.

Alake closes by distinguishing changes in token space, where remembered information can enter a later prompt, from changes in weight space, where training alters a model such as the router. He also points to reuse of internal representations through prompt caching. Of these, the router gives the most concrete meaning to owning memory. A team that captures its interactions, can say where the resulting items belong, and keeps those labels can teach the next version of its agent how to file them.