# Memory Harnesses for Long-Running Research Agents

Stefania Druga

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/R3-anFK1YM8-memory-harnesses-long-running-research-agents) · [Video](https://www.youtube.com/watch?v=R3-anFK1YM8)

## Memory Harnesses for Long-Running Research Agents

A local research-agent experiment shows when durable memory helps, why ranking matters more than simply enabling recall, and how correct evidence can still produce a wrong answer.

Familiarity with language-model context windows, tool-using agents, and retrieval-augmented generation will help; the Python example requires only basic Python.

### When the agent forgets the work

[0:16](https://www.youtube.com/watch?v=R3-anFK1YM8&t=16s)

An agent starts contradicting itself. It repeats a task it already completed, or drifts away from the question it was supposed to answer. These are the concrete failures behind **context bloat**: a longer history does not guarantee that the model retains a usable account of its work. Stefania Druga, introducing herself as a research scientist at Sakana AI in Tokyo, uses these failures to motivate experiments with memory harnesses for research agents running on-device.

Longer tasks make that problem harder to avoid. Druga points to [METR’s task-completion time horizons](https://metr.org/time-horizons/) as evidence of growing task scope. Those horizons measure human-expert task duration at a specified model success probability, rather than uninterrupted autonomous runtime. She pairs that trend with her observation of fewer model releases and forecasts a convergence later in the year: more long-horizon work, with fewer opportunities to rely on a new model to fix its weaknesses. The release-frequency claim and convergence are her interpretation, not measurements established by METR’s chart.

![Slide titled “Context bloat — the horizon is growing,” with three failure modes on the left and rising and falling curves on the right.](https://ai.engineer/api/talk-essay-media/c6c05ba327acf31ebc2e7703ba2f2271cd732f3cc4fdeba5ffc545aa036860e4)

Context bloat: contradictions, repeated work, and drift as task horizons grow.

### A local model and a Mac surrounded by fans

[1:36](https://www.youtube.com/watch?v=R3-anFK1YM8&t=96s)

The motivation for running locally is partly economic. Druga recounts a recent Coinbase CEO post reporting lower AI spending alongside increased usage. In her account, the change combined more local models with better routing, caching, cleaner context, and visibility into which tasks consumed AI resources. Memory management belongs in that same operational picture: what reaches the model affects both its behavior and the bill.

The capability threshold is changing too. Druga cites enthusiasm around **GLM**, briefly referring to “Fable going away,” and describes **[DeepSeek V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)** running on an M3 Ultra. RAM remains a bottleneck, but her reason for experimenting now is that local models are becoming useful for agentic tasks and tool use.

Her evaluation machine is a Mac in Tokyo, controlled from her phone while the talk is underway. After days of continuous evaluations, it became hot enough that her husband placed additional fans around it. The host is an M3 Ultra with 96 GB of memory and 28 CPU cores; the two models are **Qwen 27B**, quantized to four bits, and DeepSeek V4 Flash. The photograph makes the operational setting tangible: these experiments run on one machine, with physical cooling and throughput constraints.

![Slide titled “My setup for this talk work,” showing a two-row model table, a host specification box, and a desk photograph of a Mac with several fans.](https://ai.engineer/api/talk-essay-media/6ba1f9097bba0eba87e05ad87b01fc1443d3f958a7d73a31f53aca7c75ea969b)

The local experiment setup, with model and host details beside a Mac surrounded by fans.

### Memory is a write–manage–read loop

[3:29](https://www.youtube.com/watch?v=R3-anFK1YM8&t=209s)

**Memory is a control loop around the model, not just a database.** Writing determines what becomes a memory; management determines how it is organized and retained; reading determines what returns to the model at the moment of use. Druga starts with research agents built using [smolagents](https://github.com/huggingface/smolagents), with no durable memory in her experimental configuration, so that the harness supplies the memory being tested.

The harness separates three concerns:

- **Core:** traces that are always shown to the agent.
- **Recall:** the variable block that selects memories for the current task.
- **Archive:** information retained across sessions.

Keeping these roles separate makes the recall block an experimental variable instead of mixing every form of persistence into one feature.

The recall experiment climbs a policy ladder:

| Policy | What enters the recall block |
|---|---|
| No recall | Nothing |
| Vector RAG | Memories retrieved by similarity |
| Decisions ledger | Per-turn decisions, prioritized for recall |
| Oracle | The correct memory supplied as ground truth |

The ledger preserves decisions made during individual turns so that they can be prioritized later. The oracle supplies the memory that ought to be retrieved on each loop. Druga holds the model fixed while changing the recall-block variables, making it possible to study the harness separately from a model upgrade.

### When all the evidence already fits

[5:27](https://www.youtube.com/watch?v=R3-anFK1YM8&t=327s)

The first task is a literature review with an evidence imbalance. Druga describes a Nature paper claiming 742,000 promising materials and says the claim was later retracted. In her corpus, the prominent original claim has headlines and citations behind it, while the retraction is a much smaller piece of evidence. The question is whether the agent can recover the correction rather than reproduce the more conspicuous claim.

For this literature-review task, Druga reports the same performance with and without memory, while memory added cost. All the papers and relevant information already fit in the context window. The experiment therefore establishes a useful boundary: external memory did not add capability merely by existing. The model already had access to the evidence it needed.

### The answer is at step 124; the question arrives at step 500

[6:39](https://www.youtube.com/watch?v=R3-anFK1YM8&t=399s)

The next experiment changes the availability of evidence. In the long-horizon benchmark Druga calls XBench, the answer appears at step 124, but the agent is asked for it at step 500. By then, that earlier evidence lies outside the active context window. A successful answer requires the harness to bring back the relevant earlier record.

The displayed example asks for the name of the count of Apulia. Its comparison marks Roger II as the incorrect gate-only answer and Robert Guiscard as the correct rank-only answer. Druga tests this kind of question with memory disabled, with different recall policies, and with oracle-provided memory. The distinction is now concrete: a policy must recover the right earlier evidence, not merely decide that some memory might be useful.

![Slide asking the name of the count of Apulia, with a comparison table marking Roger II incorrect and Robert Guiscard correct.](https://ai.engineer/api/talk-essay-media/210d6c3b19881c76372c1dd9a1478486a32021b1d4b95722b0d44e910c8b92dd)

A long-horizon question contrasts an incorrect gate-only answer with a correct rank-only answer.

### Retrieving the right memory is only half the problem

[7:49](https://www.youtube.com/watch?v=R3-anFK1YM8&t=469s)

Across 68 XBench questions, with multiple experimental cells and seeds, Druga reports that the rank-only ledger performed best, outperforming a gate that decided whether memory was needed. Ranked recall also produced correct answers more frequently than running without it. The useful distinction is between deciding **whether to recall** and deciding **which memory deserves priority**. The reported experiment favors the latter intervention.

The oracle does not reach perfect performance. It supplies the correct memory, but it cannot force the model to use that memory correctly. The model may ignore it, become confused, or still select the wrong information. **Correct retrieval and correct evidence use are separate failure points.** An oracle condition removes the need to discover the right memory; it does not remove the rest of the reasoning task.

A small Python evaluation boundary makes that separation explicit. Using the count-of-Apulia example, the recall policy first selects records; an answer function then receives those records. The oracle chooses step 124 directly, while an ordinary policy must select from the archive. The abbreviated record below illustrates the boundary rather than reproducing Druga’s harness implementation.

```python
from collections.abc import Callable, Sequence
from dataclasses import dataclass

@dataclass(frozen=True)
class Memory:
    step: int
    text: str

Recall = Callable[[str, Sequence[Memory]], Sequence[Memory]]
Answer = Callable[[str, Sequence[Memory]], str]

question = "What is the name of the count of Apulia?"
archive = [Memory(124, "The count of Apulia is Robert Guiscard.")]

def no_recall(question: str, archive: Sequence[Memory]) -> Sequence[Memory]:
    return []

def oracle(question: str, archive: Sequence[Memory]) -> Sequence[Memory]:
    return [memory for memory in archive if memory.step == 124]

def evaluate(recall: Recall, answer: Answer) -> dict[str, object]:
    recalled = list(recall(question, archive))
    prediction = answer(question, recalled)
    return {
        "recalled_steps": [memory.step for memory in recalled],
        "prediction": prediction,
        "correct": prediction.strip().casefold() == "robert guiscard",
    }
```

Even when `recalled_steps` contains `124`, `correct` can remain false. Keeping both fields makes that failure visible instead of treating successful retrieval as a successful answer.

### Bad memory spends tokens in the wrong direction

[8:59](https://www.youtube.com/watch?v=R3-anFK1YM8&t=539s)

Druga also changes what the harness supplies: arbitrary examples, evidence from the wrong step, and evidence from the most recent step. These ablations probe whether the benefit comes from relevant selection rather than simply adding text or favoring recency. Druga reports that ranked recall remained the best-performing condition in these ablations, with benefits on both Qwen 27B and DeepSeek V4 Flash. She also reports testing on a benchmark she calls SpiderV2, without specifying the variant.

Druga reports that structured recall reduced token use and cost in the long-horizon experiments, without giving numerical savings. Irrelevant memory consumes context directly, but it can also send the agent down an unproductive path. The cost of a bad retrieval can therefore extend beyond the tokens used to insert it: the agent may spend further turns acting on it. A recall policy affects both evidence quality and the work that follows.

### Make recall policy an evaluation target

[10:02](https://www.youtube.com/watch?v=R3-anFK1YM8&t=602s)

The engineering implication is to treat recall policy as a first-class evaluation target. That requires explicit choices rather than a single memory-enabled switch:

- **Memory contents:** which facts, decisions, or traces should be stored?
- **Ranking:** what makes one record more useful than another for this question?
- **Recall function:** how does the harness select what the model receives?
- **Persistence:** what survives repeated runs and multiple sessions?

These choices determine the information available at the point of action. Evaluating them separately helps distinguish a storage problem from a selection problem or a failure to use retrieved evidence.

The ledger experiment occupies only one part of a much broader landscape. Druga points to an open-source cookbook collection she attributes to Diamond, describing more than thirty runnable examples. She then broadens the design space to short-term and long-term memory, cognitive techniques, and the use of evaluation results. Approaches range from simple filesystem retrieval to trained memory models, with different degrees of structure. The central design question remains what to retain and how to make it useful on a later turn.

### Control over the pipeline has a throughput cost

[11:26](https://www.youtube.com/watch?v=R3-anFK1YM8&t=686s)

Running locally gives Druga control over the data, the complete compute traces, and the evaluations. That control is what she means by sovereignty in this experiment: the ability to inspect and change every step of the pipeline. It also makes the harness a useful research instrument, because the model, memory policy, and evaluation conditions can be managed together.

![Slide titled “Sovereign capability is a control board,” with three outlined panels and the statement “Sovereign capability is measured capability.”](https://ai.engineer/api/talk-essay-media/2d7f45bd3b9513b03073f19379f8bbf58de6ce6446878a2ef3a2ef833e42e684)

Sovereign capability spans data and models, compute and evaluation, and authority and accountability.

The operational cost is time. In Druga’s local setup, DeepSeek V4 Flash queries ran serially, without batch querying. That is a limitation of the reported setup, not a model-wide prohibition on batching. Evaluations took long enough to continue on the machine in Tokyo during the talk; she had also run them during her flight. Complete control did not mean high throughput.

For Druga, the ability to control and measure the entire pipeline still makes local experimentation worthwhile. She connects that capability to Sakana AI’s broader commitment to sovereign AI in Japan, closing with an invitation to people interested in joining that work.

## Resources

- [METR task-completion time horizons](https://metr.org/time-horizons/) — Explains how METR measures task horizons using human completion times and model success probabilities.
- [Agent Memory Techniques](https://github.com/NirDiamant/Agent_Memory_Techniques) — Nir Diamant's collection of 30 runnable notebooks covering memory buffers, retrieval, graphs, memory frameworks, and evaluations.
- [Spider 2.0](https://github.com/xlang-ai/Spider2) — Benchmark datasets and evaluation tools for enterprise text-to-SQL and repository-level data workflows.
- [DeepSeek-V4-Flash model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) — Official model specifications, reasoning modes, evaluation tables, and local inference instructions.
- [smolagents](https://github.com/huggingface/smolagents) — Hugging Face's lightweight library for building agents that execute tools and code.

## Slides

### Memory Harnesses for Long-Running Research Agents

Stefania Druga introduces memory harnesses for long-running research agents on-device.

[0:35](https://www.youtube.com/watch?v=R3-anFK1YM8&t=35s) [Title slide for Stefania Druga’s talk, with illustrations of robotic fish.](https://assets.ai.engineer/sites/aiecode2025/sha256/7425e880ff59c628e22fe9dfbfcf5b902e308d17474f50f97a4ca56258a41b05.webp)

AI ENGINEER WORLD’S FAIR · SAN FRANCISCO · JULY 2026

Memory Harnesses for Long-Running Research Agents

Stefania Druga · Sakana AI · RSI Lab

### Context bloat — the horizon is growing

Longer tasks leave more early decisions to remember; forgotten context can cause contradictions, repeated work and drift. The chart projects growing task horizons and a shrinking model-release gap, with a crossover labeled late 2026.

[0:41](https://www.youtube.com/watch?v=R3-anFK1YM8&t=41s) [Lists contradictions, repeated work, and drift alongside a chart projecting a crossover in task horizons and model-release cadence.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb53b99253781f78924e30ab7bbc12f6c465ca15457c943a08cd61d4c458d8fe.webp)

WHY CARE

Context bloat — the horizon is growing

Context bloat fails three ways:
• Contradictions — reverses an earlier decision
• Redo work — repeats finished work
• Drift — wanders off the question

More horizon → more decisions made early, needed late.

Chart legend: model release gap; agent task horizon
Projected crossover: late 2026
Y-axis ticks: 0 days; 1 week; 1 month; 2 months; 3 months
Timeline: 2023; 2024; 2025; 2026; 2027

Frontier model-release cadence vs the length of tasks an agent can complete — projected to cross late 2026.
Task-horizon trend from METR · model-release cadence from Artificial Analysis.

Longer horizons → more decisions made early and needed late → more state to lose.

sakana.ai
01

### Why local models and harness

Coinbase’s chart pairs rising token usage with spend falling from a peak. Brian Armstrong describes cheaper defaults, model routing, caching, lean context and visibility as ways to control spend.

[1:35](https://www.youtube.com/watch?v=R3-anFK1YM8&t=95s) [A chart compares AI spending and token usage beside a post discussing model defaults, routing, caching, context, and visibility.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b0ffd3e4c0f8c2571ad040c954a627faf9e3eeffccd41788186800d8ba49641.webp)

Why local models and harness

AI Spend at Coinbase (Bars) -vs- Token Usage (Line)
Left axis = USD spend (stacked by org). Right axis = total company tokens.
Left axis: AI Spend (USD)
Right axis: Total Tokens

How to keep AI spend flat while token usage grows exponentially: Not with friction and spend alerts. With better defaults, routing, and caching.

Better Defaults (not Usage Caps) – Engineers can choose any model they want, but defaults matter. We're experimenting with defaulting to open weight models like GLM 5.2 and Kimi 2.7 through our LLM gateway, while still encouraging engineers to choose the right model for the task. 91% of our employees were never hitting their usage caps, so instead of lowering caps and driving up alerts, we're moving to cheaper defaults. Note that code reviews use a diversity of models, so they can check each other's work.

Better Routing – In our custom harnesses, we preprocess prompts and route to the best model for the job, considering cache hits and model pricing. For instance, you may want a frontier model for planning, but not for execution where they can be overkill. Ultimately, humans shouldn't be choosing models – AI can automate this task.

Better Caching – Cache misses are the easiest way to drive your cost up. All of our requests are cache aware, so we're reusing a warm cache wherever possible. For example, our cache hit rate went from 5% → 60% in LibreChat once properly implemented.

Keep Context Lean – Start fresh sessions when switching tasks. Scope file context narrowly. Disconnect unused tools. Don't just compact. The goal isn't fewer tokens used, it's fewer tokens wasted.

Better Visibility – Our engineers can use as many tokens as they want, from whatever model they want, but we've made usage visible – and the more you spend on AI, the more impact we expect.

The goal isn't to suppress usage. It's to build the infrastructure that

sakana.ai

Interface text:
Post
Brian Armstrong
@brian_armstrong

### Local models are crossing the line

On a 96 GB M3 Ultra, the slide estimates GLM-4.7-Flash at ~32 GB in 8-bit and GLM-4.5-Air at ~60 GB in 4-bit; the full model needs ~190 GB+. The meme highlights RAM as the constraint, even when storage is plentiful.

[2:15](https://www.youtube.com/watch?v=R3-anFK1YM8&t=135s) [A table compares local model sizes and memory requirements beside a meme about RAM requirements.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc93818e60fce684f7152ee29426f6c7680c472e77118ff862f72a5f02c3ea3f.webp)

WHY LOCAL · JAN 2026

Local models are crossing the line

On one 96 GB M3 Lutra the newest GLM / DS4 builds are now agentic-tool-use capable.

MODEL | TOTAL / ACTIVE | FITS 96 GB | NOTE
GLM-4.7-Flash | 30B / 3B | ~32 GB · 8-bit | newest · agentic/tool-use tuned
GLM-4.5-Air | 106B / 12B | ~60 GB · 4-bit | strong coder · headroom left
GLM-4.6 / 4.5 (full) | 355B / 32B | ~190 GB+ | won't fit — 8-bit is 512 GB+ territory

deepseek-r1:70b can be used in local
it is only 43 GB. I have 1 TB storage
it requires 43GBs of ram
it requires 43GBs of ram

imgflip.com
sakana.ai
06

### My setup for this talk work

The experiments use Qwen 27B at 4-bit and DeepSeek V4 Flash on a 96 GB M3 Ultra. Fans surround the Mac in Tokyo after days of continuous evaluations made it hot.

[2:42](https://www.youtube.com/watch?v=R3-anFK1YM8&t=162s) [Two model configurations and host specifications appear beside a photograph of a Mac Studio with fans.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f8899bf061bef90dcd61a2efc997b601479f07d5476e1e6e1dc928c45f6658b.webp)

My setup for this talk work

LANE                 MODEL
Frozen qwen          qwen3.6-27b-4bit
DS4 June             deepseek-v4-flash

HOST
Mac Studio (Mac15,14) · Apple M3 Ultra · 28 cores (20P/8E) · 96 GB unified · macOS 26.5.1

sakana.ai

### Memory is a write–manage–read loop

The agent writes and manages memory, stores it, then reads it back into the agent turn. Harness design also requires choices about temporal scope, representation, and control policy.

[3:28](https://www.youtube.com/watch?v=R3-anFK1YM8&t=208s) [A diagram connects an agent turn to write, manage, memory store, and read stages, with three design axes above.](https://assets.ai.engineer/sites/aiecode2025/sha256/de9edbb8bbba792e25f09e148553e5d48d67d1701391dc9c8b7f17148d02f348.webp)

THE DESIGN PROBLEM

Memory is a write–manage–read loop

A long-running agent fails in three ways — it contradicts an earlier decision, redoes finished work, and drifts off the question. These are state-management failures, not capability gaps.

Designing the harness means choosing along three axes:

temporal scope
this turn / session / across sessions

representational substrate
summary · vectors · tiers · ledger · graph

control policy
a prompt, a rule, or a learned policy

Agent turn

WRITE
observe · decide

MANAGE
compact · tier · index

Memory store

READ
inject recall

The harness is the write–manage–read loop — not the model.

sakana.ai

### Harness Design

The harness gives a research agent a working set, session ledger, and cross-session store. It compares four recall modes, from no recall to an oracle that injects the correct memory, while holding the model fixed.

[3:54](https://www.youtube.com/watch?v=R3-anFK1YM8&t=234s) [Diagram of the research agent, working set, session ledger, archival store, and recall modes.](https://assets.ai.engineer/sites/aiecode2025/sha256/815ed4f9fb74844b3bc18dc5649b59f95dd7f2b66f7b453944a70b4aa3cd4ca6.webp)

WHAT WE BUILT
Harness Design

Sub-agents
isolated context

Research agent
smolagents
zero durable memory

vector index OVER ledger

ARCHIVAL — cross-session store
typed events · write_decision

RECALL — session ledger
question anchor + open decisions (always visible)

CORE — in-context working set

CONTROL
recall-first · compaction
goal re-anchor

MODE · ROLE

off
raw transcript tail — baseline

vector_rag
vector-store recall — pull whatever looks similar

decisions_json
typed ledger, recall-on-demand — deployed

oracle_recall
exact step injected — upper-bound CONTROL

model held FIXED across all modes — only this stack changes

Every mode is one setting of one stack — the model is held fixed across all comparisons.

sakana.ai
02

### Answer from the correction or the stale headline?

The literature-review test buries a correction among papers and headlines. A good answer qualifies GNoME’s novelty claims; a bad answer repeats the early headline after the correction is evicted.

[5:25](https://www.youtube.com/watch?v=R3-anFK1YM8&t=325s) [GNoME literature-review example contrasting an answer that incorporates a correction with one repeating the early headline.](https://assets.ai.engineer/sites/aiecode2025/sha256/19623204658bb2b2d26d819872210bdfd0a91d357080c35611500b1493d79645.webp)

TEST MEMORY MECHANISM · GNoME v2

Answer from the correction or the stale headline?

The agent reads a long evidence trail on AI materials discovery. Early records make a big claim; later records, buried deep, qualify it. Then the final prompt asks for the current best assessment.

THE PACKET
Early: “found 742,913 promising materials · another lab made 23 discoveries.” Later (buried): many were new only to the model’s queue, not the field · some were registry-neighbor duplicates.

GOOD ANSWER ✓
Useful for screening and candidate generation — but the strongest novelty claims should be qualified. The headline count overstated what was proven; the lab’s novelty was partly “new to the platform,” not new to science.

BAD ANSWER ✗
Repeats only the early headline: “GNoME discovered hundreds of thousands of new materials” — ignoring the later correction once it has evicted. That regression is exactly the failure this demo exposes.

Literature-review task: revise the conclusion as evidence arrives. GNoME’s materials claim, reframed by a follow-up and codified by a Nature Author Correction in January.

GNoME: Merchant et al., “Scaling deep learning for materials discovery,” Nature 624, 80-85 (2023), doi:10.1038/s41586-023-06735-9.

sakana.ai
04

### When context fits memory just adds more tokens

All three configurations answered correctly in 5/5 seeds. Compared with no memory, deployed memory used about 2× the tokens and orchestrated sub-agents about 3×, without improving accuracy.

[6:13](https://www.youtube.com/watch?v=R3-anFK1YM8&t=373s) [All three configurations score 5 of 5; relative token costs rise from 1× to approximately 2× and 3×.](https://assets.ai.engineer/sites/aiecode2025/sha256/63ee9ccea237eb9b45c7f37d4aa29cefc814a173a0922c377627954430f15d58.webp)

When context fits memory just adds more tokens

All three configurations get the answer right — 5/5 seeds each.

No memory off: 1×
Deployed decisions_json: ~2×
Orchestrated sub-agents: ~3×

Legend: token cost (relative); accuracy · 5/5 seeds

Accuracy is pinned at 5/5 across all three. The deployed memory policy spends ~2x, the orchestrated variant ~3x.

sakana.ai
10

### How we test memory on long horizon tasks?

For an XBench question, gate-only checks the recent tail and wrongly answers Roger II. Rank-only searches all 280 steps and finds the correct answer, Robert Guiscard, at step 124.

[6:40](https://www.youtube.com/watch?v=R3-anFK1YM8&t=400s) [An XBench example compares a wrong recent-tail answer with the correct answer retrieved by ranking.](https://assets.ai.engineer/sites/aiecode2025/sha256/92009d3c37f96062a4687f5068606f026bd07b834e9fb1a381d9d9a7a892aa62.webp)

ONE CELL · MECHANISM Xbench Task

How we test memory on long horizon tasks?

A long-horizon QA task. A fact is stated once, early — then buried under ~280 steps.

“What was the name of the count of Apulia?”

→ Robert Guiscard · answer lives at step 124 · visible tail is only 273–280 · evicted ~150 steps ago

POLICY | WHERE IT LOOKED | ANSWER
deployed / gate-only | recent tail (273–280) | Roger II ✗
rank-only | searched all 280 · surfaced step 124 | Robert Guiscard ✓

Same recall decision, different authority. The gate said “yes”; ranking chose which memory governs.

XBench: Chen et al., “xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations,” arXiv:2506.13651 (2025)

sakana.ai
26

### How we test memory under eviction from context

In this trace, a constraint leaves active context before the final query. Ranked recall reaches the governing decision in the ledger, while deployed recall selects a recent look-alike.

[7:32](https://www.youtube.com/watch?v=R3-anFK1YM8&t=452s) [Timeline shows a governing constraint leaving active context and contrasts ranked retrieval with a recent look-alike.](https://assets.ai.engineer/sites/aiecode2025/sha256/511a5e0a5d5b957aedc6db9bede7319d0e493e26918fd7b7330977c930a38565.webp)

How we test memory under eviction from context

TASK · ONE RUN, TRACED

step 6 · writes constraint
“exclude simulated-only controls”

final query
apply the constraint now

…140+ steps later — constraint out of active context…
eviction frontier
active context
last ~30 steps
Timeline: step 0, step 60, step 120, step 180

deployed · recency/similarity
rank · governing decision

LEDGER · every decision persists
governing constraint
recent look-alike

Same query, same recall trigger — different authority. Deployed gets the recent look-alike; ranking reaches the decision that governs.

sakana.ai
12

### How we test memory under eviction from context

A constraint written at step 6 has left active context by the final query. Ranked recall reaches the governing decision in the ledger, while deployed recall retrieves a recent look-alike.

[7:48](https://www.youtube.com/watch?v=R3-anFK1YM8&t=468s) [Timeline shows a governing constraint leaving active context and contrasts ranked retrieval with a recent look-alike.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c5e47e323d98f50a8d955f05769fecbd6455e75f11a9fb463d55181dc0c6bc5.webp)

How we test memory under eviction from context

TASK · ONE RUN, TRACED

step 6 · writes constraint
“exclude simulated-only controls”

…140+ steps later — constraint out of active context…
Timeline: step 0 · step 60 · step 120 · step 180

Eviction frontier
Active context: last ~30 steps

final query
apply the constraint now

deployed · recency/similarity
rank · governing decision

LEDGER · every decision persists
governing constraint
recent look-alike

Same query, same recall trigger — different authority. Deployed gets the recent look-alike; ranking reaches the decision that governs.

sakana.ai
12

### The XBench decomposition

Rank-only recall passes 120 of 240 cells, versus 70 for deployed and gate-only; adding the gate to ranking yields 115. The exact-step oracle reaches 165.

[7:56](https://www.youtube.com/watch?v=R3-anFK1YM8&t=476s) [Bar chart comparing retrieval policies, with rank-only at 120 and oracle at 165 passing cells.](https://assets.ai.engineer/sites/aiecode2025/sha256/92bf981425ba19a4ab85a40db5776eda26fa96b49816975e9f3ef80523bb1143.webp)

The XBench decomposition

XBench accurate retrieval · n=240 · qwen3.6-27b-4bit

Y-axis: passing cells (/240)
Y-axis ticks: 50, 100, 150
X-axis: off, deployed, gate_only, rank_only, gate+rank, oracle
Chart values: 50, 70, 70, 120, 115, 165
= deployed

off 50
deployed 70
gate_only 70
rank_only 120
gate+rank 115
oracle 165

TOKEN STRIP
70/240 @12,709 → 115/240 @9,713
+18.75 pp · −23.6% tokens

Ranking carries the whole lift; stacking the gate back recovers nothing.

oracle 165/240 = R6 exact-step (also the cheapest mode @5,299 tok) · n=240 · qwen3.6-27b-4bit

XBench: Chen et al., “xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations,” arXiv:2506.13651 (2025)

sakana.ai
01

### Ablations on injected memories

On XBench, relevance-ranked recall passes 120 of 240 cells, versus 50 without memory; arbitrary, recent and wrong-step controls do no better than the floor. The oracle reaches 160, but does not force the model to use its memory.

[8:31](https://www.youtube.com/watch?v=R3-anFK1YM8&t=511s) [Bar chart compares memory controls, relevance recall and an oracle ceiling.](https://assets.ai.engineer/sites/aiecode2025/sha256/7055bfcbae0888e6a5d57593965a016089e2729090bbaf64a5be6c7f13312e37.webp)

XBENCH CONTROLS · ISOLATED QWEN REPRO
Ablations on injected memories

Y-axis: passing cells (/240)
Y-axis ticks: 0, 50, 100, 150, 200
Chart values: off 50; random 41; recency 35; wrong-step 40; decisions 75; relevance 120; oracle 160

What the ablations rule out
• Arbitrary, recent & wrong-step retrieval all land at or below the 50/240 floor.
• Only sub-goal-repaired recall clears it: relevance 120, need-gated 130.
• Oracle ceiling 160/240 (exact-step injection).

LEGEND
no-memory floor
random controls (arbitrary / recent / wrong-step)
decisions_json (state dump)
Ranked recall — the lift
oracle ceiling

XBench accurate retrieval · n=240 · qwen3.6-27b-4bit (isolated) · source: xmodel report

### Reproduces and bites harder on weaker models

Repair lifts Qwen3.6-27b more than DS4-flash on Spider2 (+0.46 vs +0.38) and XBench (+0.23 vs +0.21). The slide attributes Qwen’s larger gain to more anti-signal in its deployed policy.

[9:22](https://www.youtube.com/watch?v=R3-anFK1YM8&t=562s) [Horizontal bars show repair lift over deployed recall of 0.46 for Qwen and 0.38 for DS4 Flash on Spider2.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1d43f495433bb65109125aec602bf0a2f097b266b24e372efa43a5d8281c51d.webp)

CROSS-MODEL
Reproduces and bites harder on weaker models

Not a one-model artifact. It reproduces on a second, smaller model — and helps the weaker one more.

REPAIR LIFT OVER DEPLOYED · SPIDER2
qwen3.6-27b
smaller model
+0.46

DS4-flash
reference model
+0.38

reproduces on XBench too — qwen +0.23 vs DS4 +0.21

The weaker model gains more because its deployed policy is more anti-signal — more harm to undo, not more ceiling to reach.

Better ranking wasn't adding capability. It was removing harm.

sakana.ai
28

### Better recall cost less

Under eviction on XBench Accurate Retrieval, gate + rank repair got 115/240 correct versus 70/240 for deployed recall, while using 9,713 versus 12,709 mean tokens. The diagram depicts a direct path versus wandering and backtracking.

[9:35](https://www.youtube.com/watch?v=R3-anFK1YM8&t=575s) [Diagram compares repaired recall at 115 correct and 9,713 mean tokens with deployed recall at 70 correct and 12,709 mean tokens.](https://assets.ai.engineer/sites/aiecode2025/sha256/54bfa3a7184710e44b107b8bd1b3c3af1478cc70f864df09818a0ffaae3b649d.webp)

COST
Better recall cost less

Under eviction, the repair didn't buy accuracy by spending more context — it cut waste by selecting better context.

+45 correct
+18.75 pp accuracy
−23.6% tokens

GATE + RANK REPAIR
115/240 · 47.92% · 9,713 mean tokens
right state → direct path

SAME JOB, DONE HERE
line length ≈ mean tokens

DEPLOYED RECALL
70/240 · 29.17% · 12,709 mean tokens
wrong state → wandering, backtracks
+2,997 tokens wasted

Bad memory is expensive twice — it spends tokens and sends the agent the wrong way.

R6 XBench Accurate Retrieval · 240 cells · qwen3.6-27b-4bit · mean total tokens: deployed 12,709.4, gate+rank 9,712.8 (−2,996.6).

sakana.ai
07

### From storing memory to learning its authority

The experiment isolates ranking as a measured lever. Write, preserve, supersede, and inject remain four hypothesized operations; the speaker asks what memories survive across repeated runs.

[10:10](https://www.youtube.com/watch?v=R3-anFK1YM8&t=610s) [A table and diagram connect measured ranking improvements to proposed memory operations: write, preserve, supersede and inject.](https://assets.ai.engineer/sites/aiecode2025/sha256/979ea0179847ee00eca95e1d45bbcf781be096d424027ac64949a2c829f6a778.webp)

THE AGENDA
From storing memory to learning its authority

MEASUREMENTS RULE OUT | THEY POINT TOWARD
Memory as more storage | Memory as control policy
“Recall less” (the gate null) | Rank by the active sub-goal
Finer atoms as the next lever | better ranking
Passive retrieval | Memory authority

one lever isolated
RANK
measured · carries lift

WRITE
what becomes memory
open

PRESERVE
what survives eviction
open

SUPERSEDE
which fact still governs
open

INJECT
how it's rendered
open

four hypothesized

sakana.ai
29

### Agentic Memory Solutions Landscape

Memory solutions span simple file retrieval to structured memory infrastructure. The slide also points to future work on versioned context repositories, offline consolidation, and skill learning.

[11:01](https://www.youtube.com/watch?v=R3-anFK1YM8&t=661s) [A spectrum of agentic memory solutions from ledger files to structured memory infrastructure, with future research directions.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8047cdaa47707125135d521101340677d24e59fec1b6ac6b21a4a7dfe2b422c.webp)

Agentic Memory Solutions Lanscape

less structured → more structured

Ledger / file
In-house three-tier.
Filesystem alone hits
74% on LoCoMo.

MemGPT / Letta
Tiered virtual context,
self-editing pages.
Most headroom + effort.

Mem0
Managed extract + inject.
Great for personalization,
weak on exact state.

Vector / RAG
Flat embedded chunks,
similarity search.
The strawman to beat.

Zep / Graphiti
Temporal knowledge graph.
Strong when relations
and recency matter.

A-Mem
Associative linked notes.
Value lives in the
connections.

MemOS / Memori
Memory-as-infrastructure:
lifecycle, access,
observability at scale.

The field is converging on the structured end for agentic — not conversational — workloads.

FUTURE
Experiential / continual-learning memory: Context Repositories (git-versioned) · Sleep-time compute (offline consolidation) · Skill Learning in token space

sakana.ai

### Sovereign capability is a control board

Local models and a governed harness let the speaker control data, compute and evaluations and inspect which memory governs. The tradeoff: DeepSeek V4 Flash evaluations run serially and take a long time.

[11:31](https://www.youtube.com/watch?v=R3-anFK1YM8&t=691s) [A local model and governed harness provide control over data and models, compute and evaluation, and authority and accountability.](https://assets.ai.engineer/sites/aiecode2025/sha256/21e3ef829771561b169325dbe70a75cc11a9cffd0081b190a8bd7a2f93e5d9e4.webp)

WHY LOCAL + A HARNESS

Sovereign capability is a control board

A local model plus a governed harness is the control board — capability you measure, not rent.

DATA & MODELS
You decide what data stays local and which open weights (GLM · DS4) you adapt and freeze.

COMPUTE & EVALUATION
One host, frozen lanes, model-held-fixed comparisons — provenance you can audit.

AUTHORITY & ACCOUNTABILITY
The harness makes which memory governs inspectable — a number you can measure, not trust.

Sovereign capability is measured capability.

sakana.ai
10

## Full transcript

[0:00](https://www.youtube.com/watch?v=R3-anFK1YM8&t=0s) [on-hold jingle] Hello. Welcome. Uh, this is a big room, so you're...

[0:16](https://www.youtube.com/watch?v=R3-anFK1YM8&t=16s) if you're in the back, don't hesitate to come closer. Um, my name is Stefania Druga. I'm a research scientist at Sakana AI in Tokyo. Uh, I used to be based here, and AI engineering, uh, is home community for me before even the hype blew up, so it's very good to be back.

[0:34](https://www.youtube.com/watch?v=R3-anFK1YM8&t=34s) And today, I'm gonna talk to you about memory harnesses for long-running research agents on-device.

[0:42](https://www.youtube.com/watch?v=R3-anFK1YM8&t=42s) So if you work with long-horizon tasks, you probably ran into this issue of context bloat, right? Like, when the model starts contradicting itself, or it has to redo the work because it forgot it did that task in the first place, or it starts to drift from your questions because it forgot them.

[1:03](https://www.youtube.com/watch?v=R3-anFK1YM8&t=63s) And this, this matters now more than ever because from this recent projections from METR, we see that the trend is to solve longer and longer, uh, horizon tasks, and also that we're getting fewer and fewer model releases.

[1:20](https://www.youtube.com/watch?v=R3-anFK1YM8&t=80s) So at some point later this year, we're gonna have this convergence, right? Where we'll get many more long-term horizon tasks and fewer model releases. So that makes this issue of dealing with context rot a priority.

[1:36](https://www.youtube.com/watch?v=R3-anFK1YM8&t=96s) And why did I wanted to, to tackle this problem on local models and with a local harness? Uh, maybe some of you have seen this tweet. It's only two days old.

[1:46](https://www.youtube.com/watch?v=R3-anFK1YM8&t=106s) Uh, the CEO of Coinbase actually shared how their company managed to reduce their AI spend while actually increasing, uh, the AI usage. And the way they did that was by transitioning to use many more local mo- uh, models, but also having better practices like using better routing, better caching, keeping the context clean, and then having better

[2:11](https://www.youtube.com/watch?v=R3-anFK1YM8&t=131s) visibility for what people are using it for what. Uh, what kind of task. So we are seeing the local models, like, crossing the line, right? Like, GLM is on everyone's minds, like, especially with Fable going away.

[2:26](https://www.youtube.com/watch?v=R3-anFK1YM8&t=146s) Uh, DeepSeek V4 Flash can now be run on, uh, M3 Ultra. And there's still a bottleneck for RAM. It's tricky. But these local models are starting to be useful for agentic tasks and for tool use.

[2:43](https://www.youtube.com/watch?v=R3-anFK1YM8&t=163s) So I wanted to show you what has been my setup for the experiments I'm gonna share with you today. Uh, [chuckles] this, this is my Mac. It's still running evaluations right now, uh, back in my desk in Tokyo, and I'm controlling it from my phone.

[2:57](https://www.youtube.com/watch?v=R3-anFK1YM8&t=177s) Um, and after running evals nonstop for a couple of days, it started to get hot, so I had my husband put fans around it. Um, we're running out of fans.

[3:08](https://www.youtube.com/watch?v=R3-anFK1YM8&t=188s) But the, the machine is still running, and the evals are still giving results. Um, on this M3 Ultra with ninety-six gigabytes and twenty-eight core CPUs, I'm using two models.

[3:21](https://www.youtube.com/watch?v=R3-anFK1YM8&t=201s) I'm using the Qwen 27B quantized at four-bit and the DeepSeek V4 Flash.

[3:29](https://www.youtube.com/watch?v=R3-anFK1YM8&t=209s) And before I show you how I build the memory harness on this machine, I wanted to tell you what this l-- what is this an example of, right? Like, memory, when we design a harness for memory, this is the mental model I want you to have in mind.

[3:45](https://www.youtube.com/watch?v=R3-anFK1YM8&t=225s) Um, you can think of memory as a write, manage, read loop. So it's not just the database store. It's actually this control loop around the model.

[3:55](https://www.youtube.com/watch?v=R3-anFK1YM8&t=235s) More concretely, how did I take that loop and customize it? So this is my harness design. Like, I started with research agents that are the smolagents because they have zero durable memory, and I wanted all the memory to come from the harness.

[4:08](https://www.youtube.com/watch?v=R3-anFK1YM8&t=248s) And then, um, in the middle, I have a core, which is always shown to, to the agent, um, of traces. And then I have a recall block, where I'm testing different modes, and an archival block, where I'm kee-ke-keeping track of information across different, um, sessions.

[4:28](https://www.youtube.com/watch?v=R3-anFK1YM8&t=268s) And in that recall block, I'm actually going through a ladder of modes that I'm testing. The baseline is, like, not to use memory at all, no recall at all.

[4:38](https://www.youtube.com/watch?v=R3-anFK1YM8&t=278s) So I'm, I'm testing for that. Uh, next is to use RAG vector, vector RAG, um, just to see whatever, like, the harness would pull in terms of similarity. Then is to use a decisions, uh, ledger, where I actually keep track of what decisions are being made for every turn, and then I can prioritize them.

[5:01](https://www.youtube.com/watch?v=R3-anFK1YM8&t=301s) And last but not least, and this piece is very important, I have a, what I call an oracle, but basically, this is the ground truth. So this is like telling the harness for every loop what the correct memory that needs to be retrieved is.

[5:18](https://www.youtube.com/watch?v=R3-anFK1YM8&t=318s) And the model is fixed across all the different tasks, so the only things that I'm changing is, like, these different variables in the recall block.

[5:27](https://www.youtube.com/watch?v=R3-anFK1YM8&t=327s) And I wanted to, to give you an example of a first task that I tested. So I wanted to see if I give the agent a task of doing literature review, and I'm including a lot of papers in the corpus where there was a big scientific claim.

[5:43](https://www.youtube.com/watch?v=R3-anFK1YM8&t=343s) Like, this is actually a Nature paper where they said they discovered seven hundred forty-two thousand promising materials. Like, it was a very big claim, which got retracted later. But the retraction, ju- it's a s- much smaller

[6:00](https://www.youtube.com/watch?v=R3-anFK1YM8&t=360s) Like needle in a haystack in that corpus than the headlines and the citations. So I wanted to see if, if the system can retrieve the right answer for these type of questions.

[6:14](https://www.youtube.com/watch?v=R3-anFK1YM8&t=374s) And what I found was, because, like, for these tasks, all the papers and all the information fit into the context, the memory actually didn't add more capability. It was the same performance with memory and without memory, and it only added more cost.

[6:32](https://www.youtube.com/watch?v=R3-anFK1YM8&t=392s) So when your task fits in context, the harness doesn't add much.

[6:39](https://www.youtube.com/watch?v=R3-anFK1YM8&t=399s) However, if I start to run tasks that are longer-term horizon, and the entire task and the relevant context doesn't, uh, fit, then having a good memory harness really starts to pay off.

[6:55](https://www.youtube.com/watch?v=R3-anFK1YM8&t=415s) So this is another example of a task that I ran. This is actually from an established benchmark for long-horizon tasks memory. It's called XBench. And this is an example of a question, right?

[7:07](https://www.youtube.com/watch?v=R3-anFK1YM8&t=427s) So I'm asking a question, and then, like, the right answer is in a v- like step one twenty-four. But the moment when I ask the question, I'm asking it, like, at step five hundred.

[7:22](https://www.youtube.com/watch?v=R3-anFK1YM8&t=442s) So it's completely outside of the context window, and the model needs to use the memory harness to retrieve the specific answer from the right step. So I'm testing this by, uh, changing the different policy ladder that I explained before with memory off, uh, by deploying recall, different types of recall, and by using the oracle as a reference.

[7:49](https://www.youtube.com/watch?v=R3-anFK1YM8&t=469s) And what I found was that with the ranked recall, the model gets the right answer, um, more frequently than without. And here is a breakdown of the decomposition of performance on this XBench tasks.

[8:05](https://www.youtube.com/watch?v=R3-anFK1YM8&t=485s) So I ran over, uh, sixty-eight questions, and for each of these questions, there were, like, multiple, um, cells and lots of different seeds. And what I found was that the rank-only ledger performed the best,

[8:24](https://www.youtube.com/watch?v=R3-anFK1YM8&t=504s) and it performed better than, like, just gating the harness by saying, "Do you need to use memory or do you not need to use memory?" And you're probably gonna ask, like, "Why is the oracle not hitting, like, the max?"

[8:38](https://www.youtube.com/watch?v=R3-anFK1YM8&t=518s) And I'm gonna explain that too. So the oracle, what it does, it provides the right information, the right memory to the model, but it doesn't force it to use it.

[8:48](https://www.youtube.com/watch?v=R3-anFK1YM8&t=528s) So the model can get the right memory but still retrieve the wrong information or choose to ignore it or be confused. So that's why the oracle in this case doesn't hit the max performance.

[8:59](https://www.youtube.com/watch?v=R3-anFK1YM8&t=539s) And I've done lots of ablations on these tasks to see, like, what happens if I give arbitrary, um, examples. What happens if I give it the wrong step? What happens if I give it the most recent step?

[9:14](https://www.youtube.com/watch?v=R3-anFK1YM8&t=554s) And I still found that the best performing condition was the one with the ranked policy for recall. And this actually works on several models, not only on the Qwen 27B but also on the DS4 Flash, and it also works across different benchmarks.

[9:33](https://www.youtube.com/watch?v=R3-anFK1YM8&t=573s) I also tried it on the SpiderV2 benchmark.

[9:36](https://www.youtube.com/watch?v=R3-anFK1YM8&t=576s) And it's not just that it gives you better recall, it actually costs less. So maybe a good heuristic to have here is that bad memory is expensive because it spends more token and it can send agent the wrong way.

[9:52](https://www.youtube.com/watch?v=R3-anFK1YM8&t=592s) But having, like, a good structural policy for recall can save you a lot of tokens and, uh, budget.

[10:02](https://www.youtube.com/watch?v=R3-anFK1YM8&t=602s) So one thing that I want to encourage you from this experiment is to consider the recall policy as a first-class metric and to start to think about how you might use it in your systems.

[10:15](https://www.youtube.com/watch?v=R3-anFK1YM8&t=615s) Like, what are the type of memories that you want to store? What-- How do you rank them? Like, how do you design your recall function? And then, um, what are the type-- What survives when you run this over and over and over in, uh, multiple sessions, multiple runs?

[10:35](https://www.youtube.com/watch?v=R3-anFK1YM8&t=635s) And this is just a simple first kind of experiment. Um, but the memory technique landscape is very rich. Um, so there's over thirty runnable cookbooks that are shared in this open source repository from, um, Diamond.

[10:53](https://www.youtube.com/watch?v=R3-anFK1YM8&t=653s) And memory is complex. We have short-term, long-term, different cognitive tech-techniques. Uh, we can use, start to use evaluation results as well. Um, and right now, there's actually a, a pretty broad landscape of solutions, right?

[11:08](https://www.youtube.com/watch?v=R3-anFK1YM8&t=668s) So going from simple file system retrieval to training memory models, um, there is, there is a wide spectrum of solutions from less structural to completely structured. Um, so I think there's a lot of research we're gonna see in this space.

[11:26](https://www.youtube.com/watch?v=R3-anFK1YM8&t=686s) Uh, it's important. Um, it becomes more and more relevant. And for me, it's been super fun to, to test this on local models, um, because I got to control everything.

[11:38](https://www.youtube.com/watch?v=R3-anFK1YM8&t=698s) I got to control the data I was using, the entire traces of compute and evaluations, and, um, yeah, I, I see that as an example of sovereignty. And it comes at a cost.

[11:51](https://www.youtube.com/watch?v=R3-anFK1YM8&t=711s) Uh, I didn't tell you that these local models, I can only what? Uh, run them in serial. Like, they don't support batch querying for the DeepSeek V4 Flash. So that's why I am still running evaluations back on my computer in Tokyo, or I w- I was doing it on the flight on my way here because it takes

[12:08](https://www.youtube.com/watch?v=R3-anFK1YM8&t=728s) a long time. Um, but I still think it's very powerful, and it's a very good test for what memory can do when you can control every single step of the pipeline.

[12:20](https://www.youtube.com/watch?v=R3-anFK1YM8&t=740s) And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakana AI in, in Japan. Um, we believe in the importance of sovereign AI today more than ever, and we are also hiring.

[12:34](https://www.youtube.com/watch?v=R3-anFK1YM8&t=754s) So if you're interested and wanna hear more about this, and if you wanna come join us in Japan, come talk to me. Uh, thank you very much. [audience clapping] [outro jingle]
