AI Engineer World's Fair 2026
Active Graph Agent Runtime (BabyAGI 4)
Read the talk
ActiveGraph: building agents around their history
An immutable event log can record both what an agent does and how it changes, giving shared-state workers a foundation for memory, recovery and controlled self-modification.
From a talk by Yohei Nakajima
Before you start: Familiarity with LLM tools, agent loops and basic Python is helpful; graph runtimes and event sourcing are explained as they appear.
Why do long-running agents still break?
Long-running agents break. And if agents are becoming capable enough to build software, why must a developer keep rebuilding the agents themselves? For Yohei Nakajima, that question leads to a concrete research goal: build the simplest system that can build itself. ActiveGraph is his experimental answer—an architecture intended to make an agent’s actions and subsequent modifications inspectable.
The search began with BabyAGI, which Nakajima created in March 2023. Its public reception ran ahead of its practical capabilities; in his assessment, the original system did not work. He describes nine iterations, usually exploring some form of self-improvement. Graphs kept returning: first in Instagraph and Mindgraph for retrieval, then in representations of code, functions and logs. Alongside those experiments, he invested in agent companies through Untapped Capital and an agent fund. ActiveGraph brings that recurring graph structure into an event-sourced runtime for auditable agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The log projects the agent’s state
The architectural starting point, developed in The Log is the Agent, reverses a familiar construction process. Usually, an agent begins with an LLM, a response API, tools and memory; logging is then added around those components. That arrangement can provide the same kinds of observability and control, provided someone assembles them deliberately. ActiveGraph instead starts with the log.
The crucial addition is a record of changes to the agent, alongside the record of its work. An agent’s execution history and its development history often live in different systems. ActiveGraph puts both into one immutable event history and projects that history into a graph representing current state. A prompt might have been edited several times, for example: the log retains those changes, while a graph query retrieves the effective prompt.
Attach behaviors to that graph, and the state becomes active. A graph change triggers a behavior; the behavior emits events; those events update the graph; the updated graph may trigger another behavior. The workers can be deterministic functions or include LLM calls. They coordinate through shared state rather than talking directly to one another. The resulting typed history records both the work performed and the changes that shaped how the work was performed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Policies govern changes; events carry them
Not every change should have the same permission requirements. Policies determine how the graph may change:
| Change | Possible policy |
|---|---|
| Add a source article | Allow automatically |
| Edit a prompt | Require human approval |
| Change a fact | Check for contradictory facts |
These are different controls over the same mutation path, rather than separate mechanisms for ordinary work and agent modification.
The typed, immutable history supports replay, rollback and forks. ActiveGraph is therefore a runtime, not a prescribed agent harness: familiar harnesses can be rebuilt on top of it, provided their components communicate through shared state. Logs become the central unit to inspect, taking the place messages often occupy in an LLM-centered design.
The mutation rule is strict: query the graph, but do not edit its state directly. Emit an event instead. Even an operation that looks like adding an object must produce an event so that the graph’s new state remains derivable from its history. Otherwise, the graph could contain changes that replay cannot reconstruct.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a goal to research and a memo
Consider a goal that requires research followed by a written memo. A planner subscribes to goal creation and emits additions for two task objects—research and memo writing—and a relationship between them. The dependency itself can carry a relation behavior: once research finishes, an unblock behavior makes the memo task available. Coordination lives in the relationship and the state transitions, rather than in a direct call from one worker to another.
A small Python example makes that event flow concrete. Here, project reconstructs task state from events, and the relation behavior returns another event instead of mutating the memo task. The event names and records below are local teaching definitions, not ActiveGraph API calls.
python
from copy import deepcopy
def project(events):
tasks = {}
for event in events:
if event["kind"] == "add":
tasks[event["id"]] = deepcopy(event["value"])
elif event["kind"] == "update":
tasks[event["id"]].update(event["changes"])
return tasks
def unblock_relation(tasks, dependency):
source = tasks[dependency["from"]]
target = tasks[dependency["to"]]
if source["status"] == "done" and target["status"] == "blocked":
return [{
"kind": "update",
"id": dependency["to"],
"changes": {"status": "ready"},
}]
return []
log = [
{"kind": "add", "id": "research",
"value": {"title": "Research", "status": "ready"}},
{"kind": "add", "id": "memo",
"value": {"title": "Write memo", "status": "blocked"}},
]
dependency = {"from": "research", "to": "memo"}
log.append({"kind": "update", "id": "research",
"changes": {"status": "done"}})
state = project(log)
log.extend(unblock_relation(state, dependency))
state = project(log)
The final projection makes the memo ready while retaining the completed research task and both task titles. The history still contains the transition that caused the unblock.
Subscriptions can express more than an object type. A contradiction detector can subscribe to the creation of a claim only when that claim contradicts another claim in the graph. More elaborate conditions can incorporate confidence values. Views apply a related idea to context: a graph query selects the subset of state available to a behavior. That gives context management a programmatic boundary without preventing the use of other context-selection techniques.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Approval gates and composable packs
For protected state, an intended modification first becomes a proposed patch. Policy determines whether it needs tests, human approval or both before it can change the graph. The proposal and the applied change are different states: an agent can formulate a modification without already having permission to enact it.
A pack bundles object schemas, tools, deterministic behaviors, LLM behaviors and a pack policy. Combining packs builds a harness from both structural definitions and execution rules. This is broader than adding a skill: the module defines what objects exist, what responds to them and how changes are governed. Nakajima finds the composition unintuitive to write by hand, but reports that AI coding tools handle it well.
In the version described in the talk, event types are fixed and object types are user-defined. The current runtime documentation also permits custom application event strings alongside framework events, so that particular restriction should be read as part of the demonstrated version.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the log as memory—and as a recovery point
The first experiment asks whether the log itself can serve as memory. The retrieval method combines embeddings with the order already present in the log. On LongMemEval, the basic procedure is:
- Embed the logged messages and the incoming query.
- Retrieve messages relevant to the query.
- Include neighboring messages before and after each match.
- Fit the selected material into the available context and answer.
The initial experiment uses no semantic ingestion, fact extraction or entity extraction. It preserves conversational surroundings by using the log’s structure, rather than relying only on isolated vector matches.
The completed retrieval slide displays an 85.6% LongMemEval result. That is a displayed result, not a fully specified benchmark comparison: the talk does not establish the dataset revision, context budget or full evaluation configuration. The architectural attraction is that memory and logs already contain much of the same information; using one store avoids letting those overlapping records drift apart.
Nakajima subsequently tried semantic ingestion and reports that it improved the score, but moved on before pursuing further optimization. The more memorable operational result came during an interrupted evaluation.
In Nakajima’s account, a 500-question run lost API access around question 350; after he replaced the key, it rolled back one step and resumed around question 353. Completed work did not have to restart from the beginning. This is his reported experience of recovery during a long run, rather than a guarantee about every external tool or side effect an agent might invoke.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Coding, research and interchangeable capabilities
With the runtime in place, Nakajima asked Replit to build a coding agent on top of it. The result exposed an event log and a graph without a separate request to design those records. He explicitly acknowledges that tools such as LangSmith can provide comparable observability; his point is that the runtime made the records part of the construction process. A subsequent research agent similarly represented evidence provenance and contradictions in its graph.
To move toward broader assistants such as OpenClaw or Hermes, he assembled ActiveGraph Packs: core, tool, secret, memory, identity, communication and chat packs. Each contributes object types and behaviors. A memory pack can be replaced with another memory pack because the module includes the objects and rules that implement that capability, rather than only instructions telling an LLM how to use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Self-modification needs an acceptance test
Regimes moves from recording agent changes to controlling how those changes are proposed. A failure classifier identifies what went wrong, and that classification determines which part of the agent may be edited. The repair process therefore has a constrained target instead of permission to rewrite anything.
Nakajima recalls diagnosing failures on roughly 20 LongMemEval questions and testing a modification on 50 different questions. The published implementation specifies 50 optimization questions and 100 disjoint confirmation questions; those documented conditions should not be conflated with the exploratory loop recalled on stage. The common mechanism is to separate examples used to motivate a repair from examples used to judge it.
The proposed change follows a gated path:
- Fork the agent and propose a patch to an allowed component.
- Run a static gate check.
- Run a sandbox gate check.
- Evaluate the candidate’s effect on answers.
- Promote it only if it satisfies the acceptance policy.
The talk describes the last step as accepting only accuracy improvements. The paper’s default gate permits zero held-out accuracy change, so its documented rule is non-regression rather than strictly positive improvement. The paper also discusses over-promotion; passing a gate does not establish guaranteed cumulative progress.
Nakajima reports runs with eight or thirteen loops that accepted only four or five patches. He describes the gains as modest and statistically significant. The published evidence is narrower: significance was reported in two of five LongMemEval-S splits, with one result unadjusted for sequential promotion and pooled results treated as descriptive because the splits share a question pool. That supports a bounded experiment, not a general claim that this runtime improves every agent.
The history retains more than the promoted version. It also records proposals that failed the checks or failed to help. Rejected modifications become usable experimental memory, so a later attempt can inspect what was already tried instead of repeatedly rediscovering the same unsuccessful repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A lab that researches its own runtime
The next step is ActiveGraph Lab, a research agent that reads the project’s blog posts and repositories, develops ideas and asks Nakajima whether it may run the experiments. After approval, it runs an experiment and writes a blog post about the result. The human authorization boundary remains explicit.
Nakajima reports that the lab found an error in its own code, requested permission to fix it and wrote a pull request that he merged. It also installed a pack from another repository into itself and documented the discovery that packs were modular across repositories. These are early examples of extending and repairing the system, with approval and merging still part of the process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A deck change becomes an experiment
A Pokémon trading card game competition on Kaggle supplied a different test. The submission consists of a deck and a deterministic agent, with no LLM reasoning during matches. Nakajima describes hourly matches against new competitors that move an Elo-style score up or down. The LLM-assisted work happens outside that match runtime, while developing and evaluating candidate changes.
Using Claude Code and Replit, Nakajima reports about 80 experimental passes. A seemingly casual request—add a couple of energy cards—became a proposal to test rather than an immediate accepted edit. The example evaluation proposed 200 simulated games against three reference agents, with acceptance conditioned on win-rate improvement and a Wilson-score criterion. He does not give precise acceptance thresholds, so the example establishes the structure of the gate rather than a reproducible statistical rule.
Each pass produced a report explaining the change, its result and the verdict. Nakajima estimates that 20–30 of roughly 80 proposals were accepted, with gradual score improvement. He also mentions a figure around 27%, but does not define it well enough to identify it as a win rate or leaderboard position.
The distinctive benefit was the agent’s awareness of unsuccessful experiments. An unstructured sequence of edits can end with something that works while losing the reasons earlier attempts failed. Here, the explicit acceptance policy requires an evaluation before promotion, preserving both the successful candidates and the rejected ones. That history gives the next experiment a better starting point.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Debugging moves into the database
Across these projects, Nakajima’s personal impression is that AI coding tools are effective at designing workers that communicate through shared state. His explanation is a hypothesis: blackboard systems and Kafka have decades of architectural discussion behind them, potentially supplying more training examples than the much newer practice of building LLM agents. That is an explanation for his experience, not a measured comparison of coding performance.
A more concrete workflow change was where debugging happened. His coding agent began querying the ActiveGraph database instead of searching session logs, because the records were already cleanly typed and their structure was known. Cross-repository pack loading, recovery without restarting a long run, and access to failed experiments were other recurring benefits. Together, they made the agent’s own history part of the working development interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent needs experience as well as reasoning
The closing proposal goes beyond the runtime mechanics. With an explicit caveat that model training is not his expertise, Nakajima suggests that long-running agents need an experiential world model alongside a predictive one. A predictive model supplies priors; an experiential model retains what this particular agent has encountered and done. He compares that relationship to the hippocampus, replay, dreaming and sleep, imagining recorded experience feeding back into priors. This is a speculative analogy, not an established account of how biological memory works.
That distinction challenges the expectation that better models will make harnesses disappear. Better reasoning does not itself supply the history of a particular agent’s decisions, failures and revisions. Nakajima’s emerging hypothesis is that both are needed. People are not defined only by reasoning capability; beliefs, knowledge and behaviors also emerge from experience. An agent designed along similar lines could derive its identity from its own log.
His proposed way to explore the idea is practical: ask an existing coding agent that knows your preferences to look up ActiveGraph, build something you would find useful, and explain whether the runtime helped. The test is whether a useful implementation benefits from having its actions and evolution recorded in the same place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Runtime source, architecture overview and a recorded-fixture quickstart requiring no API key.
The architectural argument for event logs, graph projections, replay, branching and causal lineage.
Official benchmark, datasets and evaluation instructions for long-term conversational memory.
Composable assistant capabilities, example bundles and a graph inspector.
Controlled agent modification through diagnosis, constrained repairs and held-out evaluation gates.
Improvement-loop implementation, split definitions, recorded outcomes and reproduction instructions.
Research-agent project for developing ActiveGraph's evidence base.
Further reading
Evaluation harness comparing retrieval and graph-memory approaches under a shared reader and judge.
Pokémon TCG experiment tooling and a deterministic competition agent without live LLM calls.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everybody. Thanks for coming. I'm excited to be here.
- 0:16
AI Engineer World Fair has been so fun meeting everybody. Um, but I'm here to talk about ActiveGraph, which is my new open source experimental approach to building agents, which looks a little bit different than maybe you've been building agents.
- 0:29
Uh, it's definitely experimental. The idea is more to give you, inspire you with some potentially new ideas. Um, agents are awesome, but long-running agents break, and if they're so awesome, why am I still building them?
- 0:42
Why-- They should build themselves. Let's build the simplest thing that can build itself has basically been kind of my research theme for the last three years since I did BabyAGI back in March of twenty twenty-three, so that's over three years ago.
- 0:55
If you were there at the time, it was crazy. It went wild, like, uh, it was covered by media. People thought it was gonna work. It didn't work at all. [laughing]
- 1:02
Um, [laughs] uh, over the course of three years, I've done nine iterations of BabyAGI with less fanfare, but, you know, every time just experimenting on, like, how do we get autonomous agents to actually work, usually with the theme of self-improvement.
- 1:14
Uh, if you go to BabyAGI Wiki, you can see earlier experiments. Um, in th- in this process, I kept coming up to gra-- uh, coming back to graphs, and I've had a couple of projects.
- 1:23
Earlier, I did, uh, one called Instagraph and Mindgraph that was, like, pre-GraphRAG RAG. I did some code graphs, function graphs, log graphs, and since then it seems like a lot of people have started using graphs to build agents.
- 1:35
Um, and so i-in addition to that, I've actually gotten to invest in a, in a good number of, you know, agentic companies, some of which you-- I'm sure you recognize through my fund, Untapped Capital, and I also have an agent fund.
- 1:46
Uh, but yeah, that's, that's me. Yohei, VC by day, builder by night. You might recognize this face more than this face. Um, ActiveGraph is an event-sourced graph runtime for building auditable agents.
- 2:00
Um, I have a paper. That was my first archive paper called The Log is the Agent, but I'm here to explain it. Um, so today most people build agents around the LLM.
- 2:08
You start with the LLM, you add a response API, you give it tools, you add memory, and then you make sure you log everything correctly, which can give you, you know, all the benefits that ActiveGraph will give you.
- 2:19
But ActiveGraph asks, what if you built around the log? Now, what does that mean? Um, it means not e-everything the agent does, but more importantly, every change to the agent, right?
- 2:31
Nobody here is using the same agent they were using a year ago, and the agent you're gonna use a year from now is gonna be different. And a lot of people, what the agent does and how the agent changes are tracked in two different places.
- 2:42
But I'm saying let's flatten that down into a single immutable event log, and this is the ground truth of the agent. And this projects a sort of graph. This is the state of the agent.
- 2:52
And what I mean by that is, for example, a prompt can be edited multiple times, but you might have, you know, a master prompt that gets used when you're, you know, when you query the graph.
- 3:00
And then on top of this, you attach something that I'm calling behaviors. Behaviors are, uh, reacts to graph changes,
- 3:09
and then they emit events, which then in turn updates the state of the agent, which might trigger new behaviors. Um, LLMs don't talk to each other in ActiveGraph. They all communicate through this shared state, and that's what makes it a little bit different.
- 3:25
Behaviors can be deterministic, or they can include LLMs, which is, which is how you build this agent. And you get this beautiful typed event log, uh, that's the source of truth about everything the agent did and everything, every change that's happened, which means...
- 3:39
Well, actually, whoa, shoot, I j- I jumped ahead. So in, in addition to that, there's a concept called policies which determine how the graph can be modified. I'll come back to it.
- 3:46
But for example, things like a source article that you found in research, you might be fine with adding, but if you're changing a prompt, maybe you want human in the loop.
- 3:53
Or, uh, if you're changing a fact, you might wanna make sure there's no contradicting facts. So these, these thing called policies, and, and again, it shows some code, uh, code examples.
- 4:01
But yeah, in the end, you get this beautiful typed event log, which gives you replays, it gives you rollbacks, and it gives you forks, and this beco- comes natively when you start building agents with ActiveGraph.
- 4:15
So this is the kind of LLM-centric versus log-centric way of building agents that I'm gonna be talking about or showing you code for. And, and I'll, uh, I'll specify that this is not a harness.
- 4:24
It's, it's a runtime, and you can actually rebuild most of the common harnesses on top of it. You're just forcing every single communication to sh- communicate through the shared state.
- 4:34
So, uh, at the highest level, right, when you're building with agents, messages feels like the kind of core unit that you're seeing often, but we're gonna replace that with logs.
- 4:42
Logs are gonna be the core unit you're gonna build around. That's what you're gonna be reading. That's where you're gonna be looking at. These are typed logs. Again, immutable, clean.
- 4:51
You can't edit the graph. These are just kind of basic rules. Just emit events. You can have the add object, which is an event emitting. But yes, you can query over the graph.
- 4:58
And again, I'm, I'm flying through some of this 'cause I wanna get to the fun parts, and all this is in the documentation.
- 5:04
Behaviors listen to graph changes and emit events. So this is a behavior, uh, called a planner that triggers on a goal created, right? And then it, uh, adds an object, adds two task objects and a relationship object, uh, research, write memo, um, two tasks.
- 5:18
And then actually, behaviors can actually live on edges as something called a relation behavior. This one has an unblock relation, so basically, when the research is done, you can write the memo, right?
- 5:27
Um, and behavior subscriptions can be pretty complex. They can be graph queries. So this one says, "On object created, if the o- if the object type is a claim, uh, and this claim contradicts another claim, we're gonna trigger the contradiction detector."
- 5:42
And these can be m-more complex than this. You can have, you know, uh, confidence percentages baked into it. Um, and, and then on top of that, there's another concept called views.
- 5:53
Uh, context management can be done programmatically as basically a graph query. It's, uh, it's-- you basically grab a subset of the graph, which makes it available to that behavior.
- 6:02
You can still do other types of context, uh, context management, but I felt like this kind of graph query as context management just felt really elegant. Candidly, I'm not the one writing the code, but AI seems to be pretty good at figuring out how to do it.
- 6:16
Um, and you know, earlier I talked about policies. So some graph changes require a proposed patch before approval. Again, this is how you-- these, these policies kind of give it the control on what it's allowed to change by itself, what, uh, what kind of changes require certain tests.
- 6:31
Um, and I'll give a few examples in a bit, um, or if you want human-in-the-loop, right? And you have these kind of policies that, that determine or define, uh, what the, uh, or what these rules are.
- 6:42
And when you bring it all together, you get these kind of object schem-schemas, tools, deterministic behavior, LLM behaviors can be assembled into something called a pack, right, with a pack policy, and that's how you build a harness on top of ActiveGraph.
- 6:56
And these-- And all this together is, and I'll have a couple of examples later, um, are modular, and they can be combined. Uh, but you're not just adding skills.
- 7:03
It is much more complex ability. Actually pretty unintuitive. I would never write code myself with ActiveGraph, but again, AI seems really good at it.
- 7:13
Um, and just on, like, event types are fixed. You can add custom events. Objects are user-defined. I only added that 'cause someone asked me that question when I was showing them these slides, but I think that makes sense.
- 7:22
So the old way, you got the while, not done, if, loop. The new way or at least my new way, uh, you have a whole bunch of behaviors that don't talk to each other that just monitor the state.
- 7:31
So it's, it's, it's inspired by, uh, blackboard architecture from the seventies or eighties or more recently, Kafka. Whole bunch of micro workers communicating through a shared state. One of the challenges, at least back when, uh, blackboard was around was that it was really unintuitive to write, and the workers were very slim and deterministic.
- 7:46
But now AI writes the code, and the workers can be very powerful because they have reasoning capability. This is a ReAct agent on ActiveGraph. ReAct agent was one of the earlier agent kind of architectures.
- 7:58
As you can see, it's-- It look-- It actually works the same way, but on goal_created, you add a thought. On thought_created, you trigger the reason function. So again, this is just to show that you can build any harness on top of ActiveGraph.
- 8:09
It does look different, um, because they're not communicating with each other.
- 8:14
And so to see if how well this can work, I've been running a lot of experiments. Um, the first one I did was, can I use the log itself as memory?
- 8:21
So this is not pure vector RAG. It's actually leveraging the structured log, so it knows which, which message was before what, plus, uh, vec-- uh, plus embedding the actual messages within the log.
- 8:31
Uh, I did this on LongMemEval. I embedded the query. There was no semantic, uh, ingestion, no fact extraction, no entity extraction, but I just embedded the query, looked for relevant messages, grabbed a couple messages before and after, made sure it fit into the context, and it actually did pretty well on LongMemEval, right?
- 8:51
Like, a lot of the data in your memory is actually overlaps with the memory, uh, the data in your log. Actually, having them the same actually kinda makes sense and make sure they don't separate.
- 8:59
Um, I did try another couple other experiments on adding kinda semantic ingestion to improve the score. Was able to do it, but candidly, um, I could put more effort in to try to increase that, but I jumped onto the next experiment.
- 9:10
Actually, but in that process, one of the biggest fun surprises was, I don't know if you've run LongMemEval, but you have to do, like, five hundred questions. And in one of the runs, uh, my API key ran out at, like, three hundred and fifty.
- 9:20
I was like, "Oh, shoot." Okay, so I updated the API key, said, "Okay, let's, let's rerun it again." And it just, like, rolled back one and was like, "All right.
- 9:27
We'll just start from, you know, question number three hundred and fifty-three." And I don't know if you have, but, like, I've built a lot of agents that, like, where the API key did broke or something, and I had to rerun the long agent from the beginning, and that just has not been my experience since building with ActiveGraph,
- 9:40
uh, which was a very fun surprise. Uh, I built a couple reference agents, uh, right? Like, now I have this runtime. Let's see what I can build. I asked Replit to build a coding agent on top of ActiveGraph, and, and as a result, it came with, you know, event log, graph.
- 9:55
And again, I'll say this, you can do this with, you know, things like LangSmith. Thing is, I don't-- I didn't have to think about it. I just had to ask my coding agent to use ActiveGraph, and this event log and graph came natively.
- 10:05
Same thing with research agent. I just said, "Just build a deep research agent on ActiveGraph," and it came with this beautiful-- It had an event, event log and a graph of, like, where the evidence came from, what contradicts each other.
- 10:18
And again, I didn't have to think about it. I just asked my agent to build a research agent on top of ActiveGraph, and this is what it ended up looking like.
- 10:26
Um, and then I realized, okay, how do we get this closer to, like, an OpenClau or Hermes? So I, I played around. I have this thing called, uh, ActiveGraph packs, which is a core pack, a tool pack, a secret pack, a memory pack, an identity pack, a communication pack, a chat pack.
- 10:41
So now you get-- kinda get the idea of how I'm trying to build agents on top of ActiveGraph, and each of these packs have object types and behaviors. Again, feel-- probably feels very different from the way you're building agents, but I, I feel like it's actually pretty elegant.
- 10:55
You can just take a memory pack and replace it with another, another memory pack, right? Um, uh, and, and it's not like skills, but, like, the objects and the rules and all of them are, uh, bundled as a pack, and you, and you attach these packs together to create your agent.
- 11:11
Um, and then I jumped into the, the-- What I really wanted to do was start playing with self-improvement loops. Um, I did a project called Regimes. I did, uh, uh, that was in paper number two.
- 11:20
Um, this one was a really controlled self-modification. Uh, Claude Code called it Regime to Scene, but basically, we classified the type of failure, and then based on the failure it classified, it was arou-- uh, it was allowed to edit a specific part of the agent.
- 11:33
Again, I did this on LongMemEval. The loop was, I think, doing about twenty questions, looking at the answer and questions, seeing where it failed, trying to self-modify, trying that on fifty different questions, see if the accuracy actually went up, and only if it went up, it would accept it.
- 11:48
So that was kind of the proposal patch that started happening, where it would do-- after it proposed a change to itself, right? This is essentially the agent forking itself, proposing a change, doing a static gate check, a sandbox gate check, and then making sure it actually impacted the result and only then accepted a change.
- 12:05
And for these loops, it would loop, like, eight or thirteen times, but only accept four or five of those patches. And it actually did have, you know, modest, but, like, statistically significant improvement on LongMemEval scores.
- 12:16
Uh, and, and not only did it know what worked, but it also knew what didn't work.
- 12:21
Um, at this point I was like, "Okay, can I, can I get ActiveGraph to just research ActiveGraph for me?" So I s- built lab, uh, ActiveGraph Lab, which is reading all the blog posts.
- 12:30
So everything I've shared has a blog, a GitHub repo, and this lab is reading all of that to come up with new ideas, asking me if it can run it.
- 12:37
And if I say yes, it'll run the experiment, um, and then it'll write a blog post about it. It actually did find an error in its own code, asked me if it could, it could- if it could fix it.
- 12:46
It wrote the PR, and then I just merged it. Um, so it's, it's early, but it's starting to work. The lab is self-improving. Um, it also figured out that, um, it looked at ActiveGraph packs and was able to just install a pack into itself and then wrote a blog post saying, "Packs are modular between repos."
- 13:00
I was like, "I didn't know that. That's great." Um, and then I got distracted when I saw this Pokémon trading card game competition on Kaggle. As you can see, I'm all over the place.
- 13:09
Um, but, uh, you have to submit this deck in a deterministic agent, not with a, with an LLM, and they compete in this ELO-style competition. So my decks are, you know, every hour they're battling a new competitor, and their score goes up or down.
- 13:21
Um, and I felt like this was a pretty good, uh, test for seeing if ActiveGraph could help me increase my score. And I used, you know, Cloud Code and Replit, and we did about 80 different passes to like try different thing and, and increase our score.
- 13:35
Yeah, and it might not make sense to you if you don't play Pokémon, but these things like, "Let's try adding a couple energy cards," seems like a very casual ask.
- 13:42
Um, and my agent probably because, I think because I'm using ActiveGraph would say, "Okay, sounds great. Well, let's run 200 simulated games against three reference agents, and then if the win rate increases by X percent and there's a, you know, whatever Wilson score above 90-something, then we'll accept that as a change."
- 13:57
And I basically did this like 80 times, and each time they came with this like beautiful report of like why it worked, what it did, and what the verdict was.
- 14:04
So out of those 80 passes, it probably ex- ex- uh, accepted about 20 to 30, and the score did slowly improve. I'm still at like 27%, so I don't know if I can get higher than that.
- 14:13
But, um, what was most interesting is how mu- how well the agent understood experiments we've tried before that didn't work, and that was very different experience for me. 'Cause I've done a lot of YOLO agents where you just like keep trying things and then it works.
- 14:26
You're like, "Yeah." But then I don't know the stuff that we tried that didn't work. But now when I'm building with ActiveGraph, it tracks all the things that didn't work because it's forced-- because I have this policy that says, "Here are the cha- here, here's what we have to do before we accept a change."
- 14:41
Um, so some of the pleasant surprises, most of which I mentioned, but, uh, AI does seem better at architecting this in LLM-based agents is just my personal experience and, you know, some, some people, uh, you'll have to try it yourself to see.
- 14:56
But if you think about it, LLM-based agents are like three years old. But if you look at, again, Kafka, Blackboard, this like Microworker communicating through a shared state, there's decades of conversations about how to make that work better, and my hypothesis is that that's in the training data, and there's just much less training data around how to
- 15:13
build LLM-based agents. Um, debugging shifted from session logs to ActiveGraph DB. Again, I didn't know it would, but now, you know, when my coding agent was debugging, it just started querying the DB instead of the session logs.
- 15:25
K- 'cause it's everything is already logged very cleanly and typed, and it knows exactly how it's logged. Packs can be loaded easily from other repos was a surprise. I thought I had to do extra work to make that happen, but just worked.
- 15:36
Um, no more starting long runs over from the beginning, and I know what didn't work, which are some of the things I shared.
- 15:42
Um, here's where I might lose some serious researchers, at least in the language I use. I don't actually know how to train models or anything, so I'll caveat that.
- 15:50
But I'm building this, I'm starting to really think that long-running agents need not just a worl- a world model, like a predictive world model, but what I might call an experiential world model, right?
- 16:01
The predictive world model feels more like the priors, right? And then if you think about the hippocampus, actually, it also does work like an immut- uh, like a immutable state, uh, event log that projects a state, and then it feeds some of that state back into your priors through replays, dreaming, and sleep.
- 16:17
And so, you know, I feel like some, some discussions kind of suggests that as models get better, like the harness disappears. But I'm starting to think that's not true.
- 16:25
I think we need both, um, is, is kind of the new hypothesis that's growing as I, as I've been playing with ActiveGraph. And if you think about like you or me, like you or me, like we're not our reasoning capability, right?
- 16:36
We are, we're closer to our, our, our beliefs, our knowledge, and behaviors that are derived from our actual life experience. And if that's the case and we're gonna, you know, build agents inspired by ourselves, then, then maybe our agents should be treated that way too.
- 16:49
Maybe the identity of the agent is, is derived from its own log.
- 16:55
Um, I would love for you to try it. You can just go, just say, "Look up ActiveGraph and build me something I would like," to your favorite agent who knows you better than I do, uh, and have it explain if it's, uh, uh, if it was helpful or not.
- 17:07
Um, let me know if you try it or hate it or are building something relevant. Thanks for listening. [clapping] [outro jingle]