Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati

An agent that carries state over time has a harder job than answering the next question. Its earlier conversations affect what it believes, whom it trusts and what it does later. Erina Karati introduces this problem through Project Paradox, a modular framework developed at Supercell’s AI Innovation Lab for autonomous game companions that interact, compete and cooperate with players and other agents.

The companions can approach people and locations, pick up or drop objects, react to nearby events and initiate conversations. Developers can add further actions. These capabilities feed back into the agent: a conversation becomes a memory and can change its emotions, beliefs or goals. Movement therefore has a reason behind it, drawn from the character’s own remembered experience, emotion or curiosity.

Four mechanisms give that state structure:

• Separate memories: Each agent has its own memory namespace backed by retrieval-augmented generation, or RAG. One character’s stored experience does not automatically become another character’s knowledge.
• Emotion vectors: Events and conversations update a small vector containing values such as joy, sadness, fear, anger and disgust.
• Trust scores: Belief scores toward other agents and the player form a trust matrix. After an interaction, the language model decides whether a score should rise, fall or remain unchanged.
• Memory importance: The language model assigns an importance score to an event. Memories above a threshold enter a separate cache to make important context easier to retrieve later.

Importance is meant to distinguish an ordinary dinner from a shocking event that would stay with someone. This improves the chance of recovering consequential context. It does not, by itself, establish whether the memory is true, who supplied it or how confidently the agent should repeat it.

The picnic example makes the short-term behavior concrete. Asked to join a picnic, Blossom plans a sequence of actions during the conversation: pick up a pastry, then go to the picnic area. A later conversation remains in context. The request connects to a plan, the plan connects to movement and object handling, and the recent interaction remains available for the reply.

Longer interactions exposed a different problem. One agent spreads a rumor about a sale on mangoes to another, which passes it to a third. Other events intervene. When the player eventually asks about the mangoes, the answer no longer carries the expected context. The topic can survive while the history of how it became known disappears.

There are several distinct ways to fail here. The agent can forget the source, turn a rumor into a factual assertion, or remember information without using it when planning. A fluent answer about mangoes would not necessarily reveal which failure occurred. The engineering question becomes how to improve social behavior across a long run, including the consequences of earlier exchanges.

Autoresearch offers a way to turn Project Paradox into a lab bench. Define scenarios, run the agents, collect traces, score behavior and test a small policy change. The search includes how agents write and retrieve memories, communicate uncertainty, update trust, attribute sources and replan around new facts. Retrieval is one part of the protocol being optimized.

The experimental layer sits outside the village. Villagers retain local perspectives: they know what they saw, heard, remembered or inferred, and information reaches another agent through communication. The evaluator can read the full run and compare it with the scenario’s ground truth. That wider view lets it detect a source lost during retelling or a plan that ignored a known fact without giving every villager access to everyone else’s memory.

Structured traces connect the outcome to the intermediate behavior. They include observations, conversations, memory writes, retrievals and belief updates relevant to the scenario. Scoring then asks whether information spread as intended, attribution survived, uncertainty remained uncertain and agents acted on what they knew. The unit of evaluation is the entire run.

How does a proposed edit become an accepted improvement? The loop below shows the decision. A candidate changes only the controlled policy area, then faces another run. Better scores are necessary, and the guardrails must still hold; otherwise the change is reverted.

Letting agents wander can produce appealing interactions while leaving improvement hard to judge. Controlled scenarios supply a known starting condition and a specific behavioral question. The village examples exercise different parts of the protocol:

• Public fact diffusion: Agent A learns that the bakery will close tomorrow. Do the right agents learn this, retain who said it and change their plans?
• Rumor uncertainty: Agent A hears that agent C might leave the village. Does retelling preserve “might leave,” or turn it into “is leaving”?
• Replanning: A group already has a plan when one agent learns that the intended route is blocked. Does that agent communicate the change, and does the group update its plan?

Returning to the mango rumor, the observable change is modest but useful: after an autoresearch loop and a long interval, the agent answers the player’s question in context, unlike the earlier example. The causal sequence is a scenario run, trace collection and scoring, a constrained policy proposal, then a rerun. The particular edit responsible for this result is not identified. Karati explicitly withholds a claim of general improvement without repeated results; the example supports a contextual response in that run.

A single score called agent quality hides the failures that matter. The proposed scorecard separates several questions:

• Reach: How many agents know the fact after a specified number of steps?
• Source retention: Among agents who know it, how many retain where it came from?
• Uncertainty: Does an uncertain claim remain uncertain, and how often does false certainty appear?
• Planning: Are actions consistent with known information, and how long does replanning take?
• Privacy: Does information remain contained when it should?

These dimensions expose tradeoffs. Optimizing diffusion alone can encourage agents to share everything, including information they should keep private. Increasing recall can also increase the use of stale memories. A policy that improves one measure can therefore make the system worse elsewhere. The scorecard gives the experimental loop a way to catch those regressions before retaining a change.

The autoresearch layer should not be able to rewrite the entire application. Freeze the harness, scenarios and metrics, then expose the policies intended for optimization: memory writing, retrieval, communication prompts, belief and trust rules, source attribution and replanning triggers. This gives the search enough room to alter behavior while preventing it from directly rewriting the evaluation to make its own results look better.

The proposed edits target recognizable failure mechanisms:

• Attribution disappears: Preserve the source during memory writes and summarization, so compression does not discard where a claim came from.
• Rumors become facts: Store confidence, distinguish firsthand from secondhand information and require hedging when an uncertain claim is retold.
• Public facts stay local: Classify useful public facts differently and encourage agents to proactively share important source evidence.

For the mango example, these candidate policies make the missing context concrete. Remembering a sale is only part of the task; remembering its source and uncertain status allows a later answer to preserve what the agent actually learned. Because information passes through conversations, a small change to each retelling can affect behavior across the village. This is the rationale for choosing a policy space small enough to control but rich enough to change social behavior.

Adding RAG memory does not complete the design. Agents may need to preserve whether information was firsthand, secondhand, verified or uncertain. They may also need to separate raw episodic memories from current beliefs. Remembering that someone made a claim is different from deciding that the claim is true; keeping those states distinct gives later reasoning the information needed to handle uncertainty.

Rollback is part of the learning mechanism. A faster public-information policy may leak private information; stronger recall may bring outdated memories back into use. Karati describes the loop as a ratchet: try a change, score it and keep it only when the scorecard improves and guardrails hold. Reversion lets the system explore changes without treating every new policy as progress.

The proposed pattern extends to other agents whose stored state changes future decisions:

• Support agents: Track where a policy update came from and whether it supersedes an older answer.
• Personal assistants: Remember commitments and revise them when the user changes those commitments.
• Research agents: Preserve provenance and citations, handle contradictions and update hypotheses.
• Coding agents: Carry context across issues, files, teammates and changing requirements.
• Workflow agents: Respect access controls, manage handoffs and replan when conditions change.

The practical recipe is to freeze the harness, define scenarios, log traces, score behavior and expose a small policy area for search. Retain changes that survive measurement. Each application needs scenarios suited to its own state and decisions; the bakery, rumor and blocked-route tests are examples of how to make those questions concrete.

Project Paradox began with making companions feel alive in a 3D world. The deeper challenge became maintaining who knows what, who told whom, what is true or uncertain, what has become outdated and whether remembered information changes action. Those are the relationships a long-horizon experiment must exercise. The final question is concrete: across controlled runs, does the system behave better?