Memory Harnesses for Long-Running Research Agents

An agent starts contradicting itself. It repeats a task it already completed, or drifts away from the question it was supposed to answer. These are the concrete failures behind context bloat: a longer history does not guarantee that the model retains a usable account of its work. Stefania Druga, introducing herself as a research scientist at Sakana AI in Tokyo, uses these failures to motivate experiments with memory harnesses for research agents running on-device.

Longer tasks make that problem harder to avoid. Druga points to METR’s task-completion time horizons as evidence of growing task scope. Those horizons measure human-expert task duration at a specified model success probability, rather than uninterrupted autonomous runtime. She pairs that trend with her observation of fewer model releases and forecasts a convergence later in the year: more long-horizon work, with fewer opportunities to rely on a new model to fix its weaknesses. The release-frequency claim and convergence are her interpretation, not measurements established by METR’s chart.

The motivation for running locally is partly economic. Druga recounts a recent Coinbase CEO post reporting lower AI spending alongside increased usage. In her account, the change combined more local models with better routing, caching, cleaner context, and visibility into which tasks consumed AI resources. Memory management belongs in that same operational picture: what reaches the model affects both its behavior and the bill.

The capability threshold is changing too. Druga cites enthusiasm around GLM, briefly referring to “Fable going away,” and describes DeepSeek V4 Flash running on an M3 Ultra. RAM remains a bottleneck, but her reason for experimenting now is that local models are becoming useful for agentic tasks and tool use.

Her evaluation machine is a Mac in Tokyo, controlled from her phone while the talk is underway. After days of continuous evaluations, it became hot enough that her husband placed additional fans around it. The host is an M3 Ultra with 96 GB of memory and 28 CPU cores; the two models are Qwen 27B, quantized to four bits, and DeepSeek V4 Flash. The photograph makes the operational setting tangible: these experiments run on one machine, with physical cooling and throughput constraints.

Memory is a control loop around the model, not just a database. Writing determines what becomes a memory; management determines how it is organized and retained; reading determines what returns to the model at the moment of use. Druga starts with research agents built using smolagents, with no durable memory in her experimental configuration, so that the harness supplies the memory being tested.

The harness separates three concerns:

• Core: traces that are always shown to the agent.
• Recall: the variable block that selects memories for the current task.
• Archive: information retained across sessions.

Keeping these roles separate makes the recall block an experimental variable instead of mixing every form of persistence into one feature.

The recall experiment climbs a policy ladder:

The ledger preserves decisions made during individual turns so that they can be prioritized later. The oracle supplies the memory that ought to be retrieved on each loop. Druga holds the model fixed while changing the recall-block variables, making it possible to study the harness separately from a model upgrade.

The first task is a literature review with an evidence imbalance. Druga describes a Nature paper claiming 742,000 promising materials and says the claim was later retracted. In her corpus, the prominent original claim has headlines and citations behind it, while the retraction is a much smaller piece of evidence. The question is whether the agent can recover the correction rather than reproduce the more conspicuous claim.

For this literature-review task, Druga reports the same performance with and without memory, while memory added cost. All the papers and relevant information already fit in the context window. The experiment therefore establishes a useful boundary: external memory did not add capability merely by existing. The model already had access to the evidence it needed.

The next experiment changes the availability of evidence. In the long-horizon benchmark Druga calls XBench, the answer appears at step 124, but the agent is asked for it at step 500. By then, that earlier evidence lies outside the active context window. A successful answer requires the harness to bring back the relevant earlier record.

The displayed example asks for the name of the count of Apulia. Its comparison marks Roger II as the incorrect gate-only answer and Robert Guiscard as the correct rank-only answer. Druga tests this kind of question with memory disabled, with different recall policies, and with oracle-provided memory. The distinction is now concrete: a policy must recover the right earlier evidence, not merely decide that some memory might be useful.

Across 68 XBench questions, with multiple experimental cells and seeds, Druga reports that the rank-only ledger performed best, outperforming a gate that decided whether memory was needed. Ranked recall also produced correct answers more frequently than running without it. The useful distinction is between deciding whether to recall and deciding which memory deserves priority. The reported experiment favors the latter intervention.

The oracle does not reach perfect performance. It supplies the correct memory, but it cannot force the model to use that memory correctly. The model may ignore it, become confused, or still select the wrong information. Correct retrieval and correct evidence use are separate failure points. An oracle condition removes the need to discover the right memory; it does not remove the rest of the reasoning task.

A small Python evaluation boundary makes that separation explicit. Using the count-of-Apulia example, the recall policy first selects records; an answer function then receives those records. The oracle chooses step 124 directly, while an ordinary policy must select from the archive. The abbreviated record below illustrates the boundary rather than reproducing Druga’s harness implementation.

from collections.abc import Callable, Sequence
from dataclasses import dataclass

@dataclass(frozen=True)
class Memory:
    step: int
    text: str

Recall = Callable[[str, Sequence[Memory]], Sequence[Memory]]
Answer = Callable[[str, Sequence[Memory]], str]

question = "What is the name of the count of Apulia?"
archive = [Memory(124, "The count of Apulia is Robert Guiscard.")]

def no_recall(question: str, archive: Sequence[Memory]) -> Sequence[Memory]:
    return []

def oracle(question: str, archive: Sequence[Memory]) -> Sequence[Memory]:
    return [memory for memory in archive if memory.step == 124]

def evaluate(recall: Recall, answer: Answer) -> dict[str, object]:
    recalled = list(recall(question, archive))
    prediction = answer(question, recalled)
    return {
        "recalled_steps": [memory.step for memory in recalled],
        "prediction": prediction,
        "correct": prediction.strip().casefold() == "robert guiscard",
    }

Even when recalled_steps contains 124, correct can remain false. Keeping both fields makes that failure visible instead of treating successful retrieval as a successful answer.

Druga also changes what the harness supplies: arbitrary examples, evidence from the wrong step, and evidence from the most recent step. These ablations probe whether the benefit comes from relevant selection rather than simply adding text or favoring recency. Druga reports that ranked recall remained the best-performing condition in these ablations, with benefits on both Qwen 27B and DeepSeek V4 Flash. She also reports testing on a benchmark she calls SpiderV2, without specifying the variant.

Druga reports that structured recall reduced token use and cost in the long-horizon experiments, without giving numerical savings. Irrelevant memory consumes context directly, but it can also send the agent down an unproductive path. The cost of a bad retrieval can therefore extend beyond the tokens used to insert it: the agent may spend further turns acting on it. A recall policy affects both evidence quality and the work that follows.

The engineering implication is to treat recall policy as a first-class evaluation target. That requires explicit choices rather than a single memory-enabled switch:

• Memory contents: which facts, decisions, or traces should be stored?
• Ranking: what makes one record more useful than another for this question?
• Recall function: how does the harness select what the model receives?
• Persistence: what survives repeated runs and multiple sessions?

These choices determine the information available at the point of action. Evaluating them separately helps distinguish a storage problem from a selection problem or a failure to use retrieved evidence.

The ledger experiment occupies only one part of a much broader landscape. Druga points to an open-source cookbook collection she attributes to Diamond, describing more than thirty runnable examples. She then broadens the design space to short-term and long-term memory, cognitive techniques, and the use of evaluation results. Approaches range from simple filesystem retrieval to trained memory models, with different degrees of structure. The central design question remains what to retain and how to make it useful on a later turn.

Running locally gives Druga control over the data, the complete compute traces, and the evaluations. That control is what she means by sovereignty in this experiment: the ability to inspect and change every step of the pipeline. It also makes the harness a useful research instrument, because the model, memory policy, and evaluation conditions can be managed together.

The operational cost is time. In Druga’s local setup, DeepSeek V4 Flash queries ran serially, without batch querying. That is a limitation of the reported setup, not a model-wide prohibition on batching. Evaluations took long enough to continue on the machine in Tokyo during the talk; she had also run them during her flight. Complete control did not mean high throughput.

For Druga, the ability to control and measure the entire pipeline still makes local experimentation worthwhile. She connects that capability to Sakana AI’s broader commitment to sovereign AI in Japan, closing with an invitation to people interested in joining that work.