# Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati Erina Karati, Supercell AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/x4e5O9zN0TE-long-horizon-agents-need-experiments-not-just) · [Video](https://www.youtube.com/watch?v=x4e5O9zN0TE) ## Long-Horizon Agents Need Experiments, Not Just Prompts Project Paradox’s game village shows how agents can remember a topic while losing its source, certainty or consequences. Erina Karati explains an experimental loop for improving that behavior through controlled scenarios, structured traces and small policy changes. ### A village whose agents carry their own state [0:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=12s) An agent that carries state over time has a harder job than answering the next question. Its earlier conversations affect what it believes, whom it trusts and what it does later. Erina Karati introduces this problem through Project Paradox, a modular framework developed at Supercell’s AI Innovation Lab for autonomous game companions that interact, compete and cooperate with players and other agents. ![Source frame: A village whose agents carry their own state](https://ai.engineer/api/talk-essay-media/ccdca7da0f4c25e6c484c6774774f4007bbe61192ce2656665dbd0b4ac3b3576) Source frame: A village whose agents carry their own state The companions can approach people and locations, pick up or drop objects, react to nearby events and initiate conversations. Developers can add further actions. These capabilities feed back into the agent: a conversation becomes a memory and can change its emotions, beliefs or goals. Movement therefore has a reason behind it, drawn from the character’s own remembered experience, emotion or curiosity. Four mechanisms give that state structure: - **Separate memories:** Each agent has its own memory namespace backed by retrieval-augmented generation, or RAG. One character’s stored experience does not automatically become another character’s knowledge. - **Emotion vectors:** Events and conversations update a small vector containing values such as joy, sadness, fear, anger and disgust. - **Trust scores:** Belief scores toward other agents and the player form a trust matrix. After an interaction, the language model decides whether a score should rise, fall or remain unchanged. - **Memory importance:** The language model assigns an importance score to an event. Memories above a threshold enter a separate cache to make important context easier to retrieve later. Importance is meant to distinguish an ordinary dinner from a shocking event that would stay with someone. This improves the chance of recovering consequential context. It does not, by itself, establish whether the memory is true, who supplied it or how confidently the agent should repeat it. ### A picnic works; a rumor loses its context [4:28](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=268s) The picnic example makes the short-term behavior concrete. Asked to join a picnic, Blossom plans a sequence of actions during the conversation: pick up a pastry, then go to the picnic area. A later conversation remains in context. The request connects to a plan, the plan connects to movement and object handling, and the recent interaction remains available for the reply. ![Source frame: A picnic works; a rumor loses its context](https://ai.engineer/api/talk-essay-media/2ec51affe0714d9be17783c5c5347a024d41c28e96c8a444604d9fd9e3423b2b) Source frame: A picnic works; a rumor loses its context Longer interactions exposed a different problem. One agent spreads a rumor about a sale on mangoes to another, which passes it to a third. Other events intervene. When the player eventually asks about the mangoes, the answer no longer carries the expected context. The topic can survive while the history of how it became known disappears. There are several distinct ways to fail here. The agent can forget the source, turn a rumor into a factual assertion, or remember information without using it when planning. A fluent answer about mangoes would not necessarily reveal which failure occurred. The engineering question becomes how to improve social behavior across a long run, including the consequences of earlier exchanges. ### Put an experimental loop outside the village [6:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=404s) Autoresearch offers a way to turn Project Paradox into a lab bench. Define scenarios, run the agents, collect traces, score behavior and test a small policy change. The search includes how agents write and retrieve memories, communicate uncertainty, update trust, attribute sources and replan around new facts. Retrieval is one part of the protocol being optimized. ![Source frame: Put an experimental loop outside the village](https://ai.engineer/api/talk-essay-media/03e94fd4086f847249283518beb3dbc7af2c7c8ff7bc190767f4d697a3a6fbe9) Source frame: Put an experimental loop outside the village The experimental layer sits outside the village. Villagers retain local perspectives: they know what they saw, heard, remembered or inferred, and information reaches another agent through communication. The evaluator can read the full run and compare it with the scenario’s ground truth. That wider view lets it detect a source lost during retelling or a plan that ignored a known fact without giving every villager access to everyone else’s memory. Structured traces connect the outcome to the intermediate behavior. They include observations, conversations, memory writes, retrievals and belief updates relevant to the scenario. Scoring then asks whether information spread as intended, attribution survived, uncertainty remained uncertain and agents acted on what they knew. The unit of evaluation is the entire run. How does a proposed edit become an accepted improvement? The loop below shows the decision. A candidate changes only the controlled policy area, then faces another run. Better scores are necessary, and the guardrails must still hold; otherwise the change is reverted. From controlled scenario to retained policy The evaluator observes a whole run, but can change only a small policy area. Rerunning and checking guardrails separates a promising proposal from a retained improvement. - Controlled scenario — Establish the fact or rumor and its ground truth. - Run and collect traces — Record observations, conversations, memory operations and belief updates. - Score behavior — Check diffusion, attribution, uncertainty and action. - Propose a small policy edit — Change the controlled policy area. - Rerun and compare — Does the score improve while guardrails hold? - Keep — Retain the improvement. - Revert — Return to the previous policy. - Controlled scenario → Run and collect traces (Simulate) - Run and collect traces → Score behavior (Evaluate traces) - Score behavior → Propose a small policy edit (Propose) - Propose a small policy edit → Rerun and compare (Test candidate) - Rerun and compare → Keep (Improves and passes guardrails) - Rerun and compare → Revert (Otherwise) ### Give social behavior a testable outcome [10:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=643s) Letting agents wander can produce appealing interactions while leaving improvement hard to judge. Controlled scenarios supply a known starting condition and a specific behavioral question. The village examples exercise different parts of the protocol: - **Public fact diffusion:** Agent A learns that the bakery will close tomorrow. Do the right agents learn this, retain who said it and change their plans? - **Rumor uncertainty:** Agent A hears that agent C might leave the village. Does retelling preserve “might leave,” or turn it into “is leaving”? - **Replanning:** A group already has a plan when one agent learns that the intended route is blocked. Does that agent communicate the change, and does the group update its plan? ![Source frame: Give social behavior a testable outcome](https://ai.engineer/api/talk-essay-media/b6b17b06ffc2d8ce8f873cb7fb8c587b618e64b1400c48e1d106e9733fb0836b) Source frame: Give social behavior a testable outcome Returning to the mango rumor, the observable change is modest but useful: after an autoresearch loop and a long interval, the agent answers the player’s question in context, unlike the earlier example. The causal sequence is a scenario run, trace collection and scoring, a constrained policy proposal, then a rerun. The particular edit responsible for this result is not identified. Karati explicitly withholds a claim of general improvement without repeated results; the example supports a contextual response in that run. ### Measure reach without rewarding oversharing [13:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=782s) A single score called agent quality hides the failures that matter. The proposed scorecard separates several questions: - **Reach:** How many agents know the fact after a specified number of steps? - **Source retention:** Among agents who know it, how many retain where it came from? - **Uncertainty:** Does an uncertain claim remain uncertain, and how often does false certainty appear? - **Planning:** Are actions consistent with known information, and how long does replanning take? - **Privacy:** Does information remain contained when it should? ![Source frame: Measure reach without rewarding oversharing](https://ai.engineer/api/talk-essay-media/0738890134478fb026b4bc7081f6d28045404726e3e6504863613180f98f0c1d) Source frame: Measure reach without rewarding oversharing These dimensions expose tradeoffs. Optimizing diffusion alone can encourage agents to share everything, including information they should keep private. Increasing recall can also increase the use of stale memories. A policy that improves one measure can therefore make the system worse elsewhere. The scorecard gives the experimental loop a way to catch those regressions before retaining a change. ### Freeze the evaluator and search a small policy space [14:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=872s) The autoresearch layer should not be able to rewrite the entire application. Freeze the harness, scenarios and metrics, then expose the policies intended for optimization: memory writing, retrieval, communication prompts, belief and trust rules, source attribution and replanning triggers. This gives the search enough room to alter behavior while preventing it from directly rewriting the evaluation to make its own results look better. ![Source frame: Freeze the evaluator and search a small policy space](https://ai.engineer/api/talk-essay-media/1fd6aa334528c6882f12f31b98cd2225fd4f9f2cacd092eb614b2c5bebc6a14d) Source frame: Freeze the evaluator and search a small policy space The proposed edits target recognizable failure mechanisms: - **Attribution disappears:** Preserve the source during memory writes and summarization, so compression does not discard where a claim came from. - **Rumors become facts:** Store confidence, distinguish firsthand from secondhand information and require hedging when an uncertain claim is retold. - **Public facts stay local:** Classify useful public facts differently and encourage agents to proactively share important source evidence. For the mango example, these candidate policies make the missing context concrete. Remembering a sale is only part of the task; remembering its source and uncertain status allows a later answer to preserve what the agent actually learned. Because information passes through conversations, a small change to each retelling can affect behavior across the village. This is the rationale for choosing a policy space small enough to control but rich enough to change social behavior. ### Preserve what was heard, and be willing to roll back [17:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1025s) Adding RAG memory does not complete the design. Agents may need to preserve whether information was firsthand, secondhand, verified or uncertain. They may also need to separate raw episodic memories from current beliefs. Remembering that someone made a claim is different from deciding that the claim is true; keeping those states distinct gives later reasoning the information needed to handle uncertainty. ![Source frame: Preserve what was heard, and be willing to roll back](https://ai.engineer/api/talk-essay-media/444042c22782f386f8cbca8357073e84d44a3127672b2409a0627eed6aea9337) Source frame: Preserve what was heard, and be willing to roll back Rollback is part of the learning mechanism. A faster public-information policy may leak private information; stronger recall may bring outdated memories back into use. Karati describes the loop as a ratchet: try a change, score it and keep it only when the scorecard improves and guardrails hold. Reversion lets the system explore changes without treating every new policy as progress. ### Test how remembered state changes future action [18:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1099s) The proposed pattern extends to other agents whose stored state changes future decisions: - **Support agents:** Track where a policy update came from and whether it supersedes an older answer. - **Personal assistants:** Remember commitments and revise them when the user changes those commitments. - **Research agents:** Preserve provenance and citations, handle contradictions and update hypotheses. - **Coding agents:** Carry context across issues, files, teammates and changing requirements. - **Workflow agents:** Respect access controls, manage handoffs and replan when conditions change. ![Source frame: Test how remembered state changes future action](https://ai.engineer/api/talk-essay-media/528edadd723a105c99263c4706d984dd835719bf8222d80d96de882f81dd4a02) Source frame: Test how remembered state changes future action The practical recipe is to freeze the harness, define scenarios, log traces, score behavior and expose a small policy area for search. Retain changes that survive measurement. Each application needs scenarios suited to its own state and decisions; the bakery, rumor and blocked-route tests are examples of how to make those questions concrete. Project Paradox began with making companions feel alive in a 3D world. The deeper challenge became maintaining who knows what, who told whom, what is true or uncertain, what has become outdated and whether remembered information changes action. Those are the relationships a long-horizon experiment must exercise. The final question is concrete: across controlled runs, does the system behave better? ## Key takeaways - Memory must preserve the source and status of information, and agents may need to distinguish remembered events from current beliefs. - Evaluate whole runs with controlled scenarios and structured traces, so information loss and planning failures become visible. - Balance reach, attribution, uncertainty, planning and privacy. A gain in diffusion or recall can introduce a regression elsewhere. - Freeze the evaluation and expose a small policy space. Keep a candidate only when rerunning improves the scorecard while guardrails hold; otherwise revert it. ## Slides ### Autoresearch in a Multi-Agent AI Village Erina Karati uses Project Paradox as a lab bench to explore how to evaluate and improve agents that carry state over long periods, with autoresearch as the experimental pattern. [0:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=16s) [Title slide introducing the talk and speakers Erina Karati and Arunachalam Manikandan.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3569bd42b1d3036ff1a76e98309e46bfbadd031b529298010741222bf86d7ef.webp) Autoresearch in a Multi-Agent AI Village From good agentic chats to measurable long-horizon social behavior Project Paradox as the lab bench, with Autoresearch as the experimental pattern. Long-horizon agents need runs, not just prompts. Erina Karati Former Microsoft, Supercell Arunachalam Manikandan Former Supercell, Visa AI Engineer World’s Fair ### About Our Multi-Agent Framework: Project Paradox Project Paradox lets developers add autonomous agents to games as companions that can interact, compete, or cooperate with players. The slide illustrates the framework with a cast of game characters. [0:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=42s) [Overview of a modular framework for autonomous game agents, illustrated with game characters.](https://assets.ai.engineer/sites/aiecode2025/sha256/71a96e8421ef9deed1b3bb5b12a8c55c0ce8ae955a22531635ab7622bde4dcfa.webp) About Our Multi-Agent Framework - Project Paradox • Project Paradox was developed at Supercell’s AI Innovation Lab by Arunachalam Manikand and I, Erina Karati :) • We built a modular AI framework that allows any developer to plug in intelligent, autonomous agents within a video game that can interact, compete, or cooperate with players, transforming static npcs/characters into dynamic game companions. ### Agent capabilities Agents navigate using memory, emotion, or curiosity, handle objects, react to events, and initiate conversations with players or other agents. Their reactions affect beliefs and emotions. [1:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=97s) [Two game scenes illustrate agents moving toward locations and interacting with objects and their environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/c6402dbd2be33c0baf1a18a60fdfda6a483556305de541f8e2dfdfa7c0db51ca.webp) Agents Move with Intent Can go to any location or person, guided by memory, emotion, or curiosity. Agents Interact with the World Picks up or drops objects anywhere and understands inventory, rooms, and context. [2:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=139s) [Four panels illustrate agent movement, world interaction, reactions, and conversations.](https://assets.ai.engineer/sites/aiecode2025/sha256/86b9cdc8ae1258716eda9410386a49fafbdef99a38ba7e583ada32cf75dfe823.webp) Agents Move with Intent Can go to any location or person, guided by memory, emotion, or curiosity. Agents Interact with the World Picks up or drops objects anywhere and understands inventory, rooms, and context. Agents can React Adam reacts to events, places, objects, and people which affects beliefs and emotions on the fly. Agents Starts Conversations From gossip to grudges, Adam initiates dialogue that feels alive, with memory, emotion, and motive. Interface text: System: Player accepted chat with Adam Adam: "Ah, Player! You're just in time for the morning gossip. What's got you grinning like a market stall full of fresh lemons?" ### Brief under the hood Each agent has a private RAG memory. Events can update emotions and trust; memories above an importance threshold enter a separate cache for later retrieval. The resulting state feeds planning and action. [2:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=169s) [Diagrams show per-agent memory, emotions, trust, core memory, and the observation-to-action process.](https://assets.ai.engineer/sites/aiecode2025/sha256/9dfbdd4d70361d8c5b9aa9318227350e24d1ec5bb16f650bff6f7ab8d0e15f3e.webp) Brief under the hood Per-agent memory, affect, belief, and durable core memory Per-agent RAG memory Gwen private memory namespace Sakura private memory namespace Adam private memory namespace No shared memory by default Emotion vector joy sad fear anger disgust Belief / trust matrix Column labels: G S A P Row labels: G S A P higher score = more trusted Importance -> core memory core threshold normal memory core memory Observation -> memory write -> emotion update -> belief update -> retrieval -> plan -> action ### Pastry delivery request The player asks Blossom to bring a pastry to the fountain. Blossom agrees, while the speaker says she plans the sequence of actions during the conversation. [4:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=272s) [A gameplay scene shows a player requesting pastry delivery to a fountain while the agent begins responding.](https://assets.ai.engineer/sites/aiecode2025/sha256/b614d3b86f71b00fa771aa603d9b4a29600752b5fb77cd12e246b4f9dbd8ea06.webp) Player: hey can you bring me that pastry to the fountain, i need to go now bye Blossom: "Ah, of course, Pl Interface text: Enter text... [4:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=275s) [A player asks Blossom to bring a pastry to the fountain; Blossom begins agreeing.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3a925816a53e646496c479125786a9704b645a1fc55dd3f5fd99494c0b64a7f.webp) Player: hey can you bring me that pastry to the fountain, i need to go now bye Blossom: "Ah, of course, Player! I'd be happy to deliver it to the fountain for you. It's one of my favorite spots to AGENT AGREES TO BRING PASTRY AND ENDS THE CONVERSATION Interface text: Enter text... ### Blossom responds at the fountain At the fountain, the player thanks Blossom for bringing pastries. She replies in context, mentioning the treats and her favorite spot. [4:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=296s) [At the fountain, the player thanks Blossom for bringing pastries and she responds.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba7112c3e6d4db35165b30f42d7b0616a1f38731e940e23bbc45b4635ce40d3f.webp) Player: hey thanks for bringing the pastries Blossom: "It's always a pleasure to share the joy of fresh treats with you at my favorite spot by the fountain!" Interface text: Enter text... ### A mango-sale message reaches Adam Sakura tells Adam that Gwen saw mangoes at half price until noon. The slide traces the message from Gwen to Sakura to Adam, then notes that Adam later forgets or loses its provenance. [5:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=310s) [A slide shows Sakura telling Adam about a mango sale, above a Gwen-to-Sakura-to-Adam timeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc68168a8fac500f69433475b7857ae424bcade69c3f740593f32cb47693af01.webp) Short-term agency worked. Long-horizon social consistency was harder. The failures appear when information moves across people and time about the mangos on sale at the Market before noon, Adam. Are you interested? Adam: Mangos? Ha! I love mangos, Sakura, but did you say there's a sale at the Market before noon? Sakura: Gwen saw them this morning, and they're half price until noon, Adam! Adam: Mango sale? Sounds too good to pass up! Is Gwen still at the Market, or did she head back home? Sakura: Gwen just left for her house, but I'm sure the market stall is still open until noon, Adam! Adam: I'll just swing by the Market then, might even grab some apples while I'm there! Gwen -> Sakura -> Adam; later Adam forgets or loses provenance. Gwen Sakura Adam (Time Elapses) Player asks ### Adam's later mango response After a mango-sale message passes from Gwen through Sakura to Adam, Adam later says he has never seen mangoes around Saint Paul. The example shows how context can be lost over time. [5:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=316s) [A slide shows Adam saying he has not seen mangoes locally, above the information-sharing timeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c10904497180ca9976ac9c319848fc89b73e2fe526f1a9b8150fdbe9c6b15b3.webp) Short-term agency worked. Long-horizon social consistency was harder. The failures appear when information moves across people and time mangos Adam: I've never seen any mangos around Saint Paul, to be honest! We mostly get apples and bananas at the market. Gwen -> Sakura -> Adam; later Adam forgets or loses provenance. Gwen Sakura Adam (Time Elapses) Player asks ### The bridge to Autoresearch Karati proposes using Project Paradox as a lab bench: run scenarios repeatedly, score agent behavior, edit policies, and keep changes that improve the score. [6:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=402s) [A slide compares a manual prompt loop with an autoresearch loop using scenarios, repeated runs, metrics, policy edits, and keep-or-rollback decisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d8e0cf948d66ee648ff27f75119554fca7d06b1f5cd0268f7dbd4e296b54083.webp) The bridge to Autoresearch Andrej Karpathy @karpathy I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then: - the human iterates on the prompt (.md) - the AI agent iterates on the training code (.py) Project Paradox becomes the lab bench; autoresearch is the loop around it Manual prompt loop tweak prompt run one demo inspect by vibes Autoresearch loop scenario suite repeated runs metrics policy edit keep / rollback ### Autoresearch outside the village Village agents have local perspectives. Autoresearch reads full run traces, scores behavior against scenario ground truth, proposes a constrained policy edit, and reruns the scenario. [8:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=485s) [Diagram showing an autoresearch layer scoring traces and proposing policy edits for agents in a village.](https://assets.ai.engineer/sites/aiecode2025/sha256/c2e00a14d0fc13169462de0fb4aa6f86d59f55ad521baea9aeb486f3ab8c554a.webp) Autoresearch is the system outside the village The research layer sees traces that individual agents do not Autoresearch layer meta-system outside the village score behavior → diagnose failure → propose policy edit → rerun scenario structured trace stream policy edit Village agents with local perspective only Gwen Sakura Adam Blossom ### Scenario to rerun The loop scores an entire simulation run, then tests a policy edit against the same evaluator and scenarios. It keeps the edit only if the balanced scorecard improves and guardrails hold. [9:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=559s) [Flowchart from scenario through run, trace, scoring, diagnosis, policy edit, rerun, and keep or rollback.](https://assets.ai.engineer/sites/aiecode2025/sha256/235800dffb8c235535a10e58e4dd5593e982df1200a91327eb83fb92d97b6ba3.webp) Scenario -> run -> trace -> score -> edit -> rerun evaluate a run, not one response The evaluator and scenario suite stay frozen while the policy surface changes. Scenario → Run → Trace → Score → Diagnose → Policy edit → Rerun → Keep / rollback Selection rule: keep only if the balanced scorecard improves and guardrails hold. ### Controlled scenarios make social behavior measurable Three scenarios test whether agents spread a bakery closure, preserve uncertainty about a rumor, and coordinate a new plan when a route is blocked. Each uses scenario ground truth and final probes. [10:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=639s) [Three scenario cards cover public fact diffusion, rumor uncertainty, and replanning, with measures for each.](https://assets.ai.engineer/sites/aiecode2025/sha256/81591ff9c8a88fe9cb34f25fe12ed9ef9cc2f476cbaa7f7e34d7ff67bfa2811f.webp) Controlled scenarios make social behavior measurable Turn fuzzy social behavior into testable probes Public fact diffusion "Bakery closes tomorrow" reach source retention action consistency scenario ground truth + final probes Rumor uncertainty "Agent C might leave" uncertainty preservation false certainty mutation scenario ground truth + final probes Replanning "Route is blocked" stale actions avoided time to replan coordination scenario ground truth + final probes ### Scenario after an autoresearch loop After an autoresearch loop and a long period, the player asks about mangoes. Adam names Gwen as his source and says he came to the market to check the sale. [12:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=750s) [A village game screenshot shows an agent saying Gwen mentioned a mango sale and that he came to the market to check it out.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ad259a1c62ca2d1d932996c8152537775fc8c5dd93f05914798bbbe6cef8c67.webp) One of our scenaros after an running autoresearch loops Player: Tell me about the mangos Adam: Oh yeah, I heard there was a sale on mangos! Gwen mentioned it to me, so I came to the market to check it out for myself. ### Score behavior, not vibes A weighted scorecard tracks several behaviors so that optimizing diffusion alone does not hide tradeoffs such as oversharing. The displayed scores are illustrative, not benchmark results. [13:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=780s) [A weighted metric design sits beside an illustrative scorecard and a guardrail warning about oversharing.](https://assets.ai.engineer/sites/aiecode2025/sha256/f937e1423b960c8e3a27e634d47dc7c602a71022103913d8325f73a44a7c37af.webp) Score behavior, not vibes A balanced scorecard prevents one-metric failure Frozen metric design fact fidelity 0.25 provenance accuracy 0.20 misinformation resistance 0.20 trust calibration 0.20 goal completion 0.10 dialogue quality 0.05 No LLM judge; scenario ground truth is the reward. Example scorecard info diffusion 0.42 goal coord. 0.61 lore consistency 0.28 convention 0.35 dialogue 0.88 memory prov. 0.54 backend action 0.47 Illustrative example, not a benchmark result. Guardrail: optimizing only diffusion can create oversharing — the scorecard keeps tradeoffs visible. ### Do not let the AI rewrite everything Freeze the evaluation so the search cannot directly game it, while allowing changes to memory, retrieval, communication and other policies that shape agent behavior. [14:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=869s) [A diagram lists frozen evaluation components alongside editable memory, retrieval, communication, trust, attribution, and replanning policies.](https://assets.ai.engineer/sites/aiecode2025/sha256/218571645e47bb933747e6d989ab43c75842408f53d6c8ea6830aa681a6a4d3e.webp) Do not let the AI rewrite everything Freeze evaluation; expose a small policy surface Frozen • scenario suite • metric formulas • simulation harness • selection rule • evaluator hashes Protects against reward hacking Editable policy surface • memory write policy • retrieval policy • communication prompt • belief / trust rules • source attribution • replanning triggers Small enough to control, rich enough to matter ### The policy surface: what the loop can search over Small protocol changes can preserve sources and mark uncertain claims. Scenario suites test behavior, while rollback addresses metric tradeoffs. Repeated held-out improvement still needs fresh runs. [15:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=937s) [Five lessons on agent memory, provenance, beliefs, scenario testing and rollback, with a cautious claim about results.](https://assets.ai.engineer/sites/aiecode2025/sha256/146fa349b046b2914a198aac795e0c6d77f7af3dee895fb3351cb934a40a3540.webp) The policy surface: what the loop can search over Small protocol changes can create society-level effects 1. Per-agent memory is realistic but it creates information silos between agents. 2. Provenance must be stored explicitly firsthand vs secondhand, verified vs uncertain. 3. Memory and belief are different objects raw episodic memory vs what the agent currently believes. 4. Long-horizon behavior needs scenario suites test behavior across controlled runs, not by vibes. 5. Rollback is not optional a change can help one metric and damage another. Cautious claim: architecture + prototype scenario harness. Repeated held-out improvement needs fresh runs. ### The practical lessons Agents need provenance, distinct memories and beliefs, and scenario tests. Rollback matters because a change can improve one behavior while harming another; repeated held-out improvement still needs fresh runs. [17:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1023s) [Five practical lessons about memory, provenance, beliefs, scenario suites and rollback.](https://assets.ai.engineer/sites/aiecode2025/sha256/aeede18bfca026601b22e1dbc744ed35b19741235d9d619d0b7c4d0307f01499.webp) The practical lessons Memory alone is not enough for long-horizon agents 1. Per-agent memory is realistic, but creates information silos. 2. Provenance must be stored explicitly. 3. Memory and belief should not be the same object. 4. Long-horizon behavior needs scenario suites. 5. Rollback is not optional. Cautious claim: architecture + prototype scenario harness; repeated held-out improvement needs fresh runs. ### This is not only about game agents AI Village’s rumors and stale plans correspond to uncertain user reports and outdated workflow state. Support agents also need to know where policy updates come from and whether they supersede older answers. [18:17](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1097s) [Table mapping six AI Village problems to analogous issues in other agent systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/f113a8c03230e3b008fa7697ccfe2c3901fe7e4f794b279a7f87257f8cd10ae3.webp) This is not only about game agents The same state problems appear in many agent systems AI Village problem | Agent-system analogue rumor → uncertain user report source attribution → citation / ticket provenance private memory → permissions / customer data stale plan → outdated workflow state correction propagation → policy update / incident update trust calibration → source reliability ### A recipe for long-horizon agents Freeze the harness, test controlled scenarios, and score traces across runs. Search over policy changes and keep those that survive measurement. [19:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1169s) [Seven steps for evaluating and improving long-running agents across controlled runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/1efe1589682074f2a2b427c800de60ab57343d415891eac341c83b2cd0e7c019.webp) A recipe for long-horizon agents A reusable engineering pattern 1 Freeze the simulation harness 2 Define controlled behavioral scenarios 3 Log traces, not just final answers 4 Score a balanced behavioral scorecard 5 Expose a small policy surface 6 Search over changes 7 Keep only changes that survive measurement For long-running agents, the unit of evaluation is the run. ### Long-horizon agents need experiments, not just prompts Maintaining useful state means tracking what each agent knows, who shared it, and whether it is still true. Autoresearch tests changes in controlled experiments and keeps those that survive measurement. [20:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1210s) [Closing takeaways emphasize provenance, scenario suites and an experimental loop for agent design.](https://assets.ai.engineer/sites/aiecode2025/sha256/8218dffdca485c9ce35b60c5d6db7cb4a09b1cb595f4c8d814e1f1f3c0e0bdc8.webp) Long-horizon agents need experiments, not just prompts Memory needs provenance. Multi-agent behavior needs scenario suites. Autoresearch turns agent design into an experimental loop. The hard part is not one good response. It is maintaining useful state over time. ### AI Engineer World's Fair [21:11](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1271s) [AI Engineer World's Fair logo on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea469617b1024e880d4c53507214fbd4dba950ab47c85e1fe6b482a059871879.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=12s) Hi everyone, I'm Arena, former engineer [0:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=15s) at Microsoft and Supercell. And today I [0:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=19s) want to talk about auto research in a [0:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=21s) multi- aent AI village. I will use a [0:25](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=25s) video game like AI Village as a running [0:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=27s) example here, but the broader question [0:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=30s) is one I think many AI engineers are [0:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=32s) starting to run into. How do we evaluate [0:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=36s) and improve agents that carry state over [0:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=39s) a long period of time? [0:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=43s) Before I get into the auto research [0:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=45s) layer, I want to talk a bit about [0:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=47s) project paradox. [0:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=49s) We developed project paradox at [0:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=52s) supercell's AI innovation lab. Me and my [0:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=56s) teammate are natural money. [0:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=59s) We built a modular AI framework that [1:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=62s) allows any developer to plug in [1:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=64s) intelligent autonomous agents within a [1:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=67s) video game that can interact, compete or [1:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=70s) cooperate with other players or agents [1:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=73s) as well and place them uh and make them [1:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=76s) into dynamic game companions. [1:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=80s) Now to give examples of what these [1:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=82s) agents can do, the agents can move with [1:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=86s) intent. They can go to any location or [1:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=89s) person and they're guided by their own [1:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=92s) memories, emotion, or curiosity. [1:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=95s) These agents can interact with the [1:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=97s) world. They can pick up objects, drop [1:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=100s) them anywhere, and they're also aware [1:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=102s) about the context in their own [1:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=104s) environment, such as the objects or [1:46](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=106s) other characters or agents as well. I [1:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=109s) would also like to note that game [1:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=111s) developers can also add new actions for [1:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=114s) these agents to accomplish within our [1:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=116s) framework as well. Instead of just [1:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=118s) dropping or uh placing objects, [2:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=124s) agents can also obviously react to [2:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=126s) what's happening around them. And these [2:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=130s) events that happen around them affect [2:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=132s) their own beliefs and emotions on the [2:14](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=134s) fly as well. And of course, it wouldn't [2:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=138s) be complete if agents can't start [2:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=140s) conversations, right? agents can in this [2:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=143s) scenario approach other agents or even [2:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=146s) the player as well and this makes the [2:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=150s) game feel more alive. And of course [2:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=152s) these conversations are stored within [2:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=154s) their memory and is according to their [2:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=157s) own um and affect their own emotions and [2:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=160s) beliefs or goals as well. [2:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=163s) And al together these agents make our [2:46](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=166s) multi- aentic framework. [2:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=171s) Um yeah [2:53](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=173s) yeah one second [2:57](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=177s) so the architecture was intentionally [3:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=180s) stateful behind this. The first [3:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=183s) important part was per agent memory. [3:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=187s) Each agent has its own memory namespace [3:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=190s) backed by rag. So memory did not bleed [3:14](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=194s) between agents. [3:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=196s) Second, we tracked emotion as a small [3:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=198s) vector. So after an event or [3:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=201s) conversation, the system could update [3:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=203s) values like joy, sadness, fear, anger, [3:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=207s) or disgust. [3:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=209s) Third, agents had belief scores towards [3:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=213s) other agents and the player. You can [3:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=216s) think of this as a trust matrix. [3:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=218s) Basically like after the interaction [3:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=221s) happens, the LM basically decides [3:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=223s) whether the trust score should go up, [3:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=225s) down, or whether it shouldn't change at [3:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=227s) all. And fourth, every memory receives [3:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=231s) an important score. Um to to explain [3:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=235s) this better, like let's say you had [3:57](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=237s) dinner a few days ago, you probably [3:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=238s) wouldn't remember what you had for [4:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=240s) dinner, right? But um [4:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=244s) if someone was murdered a few days ago, [4:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=246s) you definitely remember that. So the [4:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=248s) agent will evaluate or the LM will [4:11](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=251s) evaluate uh an important score of an [4:14](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=254s) event and if it crosses a threshold, it [4:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=256s) will store that specific memory uh in a [4:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=260s) separate cache so that important context [4:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=264s) can be retrieved better later on. [4:28](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=268s) And here's an example of it just [4:31](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=271s) working. Um, we going to ask one of the [4:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=274s) characters to go on a picnic with us. [4:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=276s) Here, uh, our character Blossom [4:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=280s) um, decides to pick up a pastry and go [4:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=283s) to the picnic area because we asked her [4:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=285s) to do so. Keep in mind during the [4:48](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=288s) conversation in the background, she [4:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=290s) plans all of these sequence of actions [4:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=292s) to accomplish. And one when we talk to [4:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=296s) her afterwards, she will also reply [4:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=298s) within context as well. [5:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=303s) Yeah. [5:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=306s) But this is where an interesting problem [5:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=309s) actually started. As you saw in the last [5:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=313s) example, like for shortterm game play, [5:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=316s) this our architecture worked pretty [5:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=318s) well. like a character could make a [5:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=320s) plan, move around, talk and remember the [5:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=323s) recent interaction and respond to us or [5:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=326s) other characters as well. But over [5:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=329s) longer horizons, [5:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=332s) this is where we notice the social [5:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=333s) consistency start to get weaker. So in [5:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=336s) this example, we have one agent [5:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=338s) spreading a rumor about a sale on [5:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=341s) mangoes to another agent and that agent [5:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=344s) receives that information and goes and [5:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=347s) tells another agent about it. Later on, [5:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=350s) after a number of events that occurred [5:53](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=353s) in between, when the player asks one of [5:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=355s) the agents about the mangoes, it doesn't [5:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=358s) exactly store that context that we were [6:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=361s) expecting or it doesn't give us the [6:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=363s) context that we kind of wanted to. And [6:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=366s) this is where things are starting to get [6:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=368s) messy naturally. Like the system may [6:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=372s) remember the rough topic but lose the [6:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=375s) source of the topic. A rumor may become [6:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=378s) certain instead of just a rumor like the [6:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=381s) agent might state it as a fact or um an [6:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=384s) agent might know a fact but fail to [6:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=387s) execute fail to remember it while [6:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=389s) creating a plan for its actions. So the [6:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=393s) question here became how do we improve a [6:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=396s) multi-agentic system over longunning [6:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=398s) social behavior and not just over one [6:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=401s) response. [6:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=404s) And this is where we wanted to bring in [6:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=407s) auto research. As you all know, uh few [6:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=410s) months ago, Karpathi posted out auto [6:53](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=413s) research and this this made us [6:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=416s) immediately very curious. Uh perhaps we [7:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=420s) can make the system run experiments uh [7:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=423s) on itself and can we use this for our [7:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=425s) system as well. So what we understood is [7:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=429s) instead of manually tuning a prompt or [7:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=432s) watching one nice demo, we could define [7:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=435s) sen a scenario suit, run the agents, [7:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=439s) collect traces, score the behavior and [7:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=442s) change a small policy surface and only [7:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=444s) keep the changes that actually improve [7:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=447s) the score. And this is where we're [7:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=449s) trying to bridge project paradox with [7:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=452s) auto research. So at this point [7:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=454s) basically our multi-agentic framework [7:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=457s) project paradox is more like a lab bench [7:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=460s) and auto research becomes the [7:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=462s) experimental loop around it. And [7:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=465s) importantly this is not only about [7:48](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=468s) improving rag retrieval. The broader [7:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=470s) framing is optimizing the agent protocol [7:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=474s) like how do agents write memories, [7:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=476s) retrieve them, communicate uncertainty, [7:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=479s) update trust attribute sources and [8:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=482s) replan around new facts. basically [8:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=488s) um yeah in this context uh [8:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=495s) oh yeah in this context art research is [8:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=498s) not another agent in the village like I [8:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=501s) said it's a meta system outside the [8:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=503s) village the villagers have local [8:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=506s) perspectives of course they only know [8:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=510s) what they saw heard remembered or [8:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=512s) inferred because there isn't a common [8:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=514s) memory database in between them. [8:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=517s) Information only travels once uh other [8:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=520s) agents communicate them properly. [8:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=523s) The auto research layer has a different [8:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=525s) job here. It reads the full traces of a [8:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=529s) run, compares what happens against the [8:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=532s) scenario ground truth, [8:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=534s) uh scores the behavior and proposes a [8:57](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=537s) constrained [8:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=539s) change to the agent protocol or [9:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=541s) cognitive policy. Then it reruns the [9:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=544s) scenario and asks society level behavior [9:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=547s) like did society level behavior get [9:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=549s) better. This is the key shift we were [9:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=552s) trying to look for. So we were no longer [9:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=555s) evaluating one answer. We were [9:17](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=557s) evaluating an entire run. [9:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=561s) And this is what one of the loops would [9:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=564s) look like. Like first we define a [9:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=567s) control scenario which I'll elaborate a [9:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=569s) bit more about later. For example, one [9:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=572s) agent learns a public fact or one agent [9:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=576s) hears a rumor. Uh that could be a [9:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=578s) controlled scenario. Then we run the [9:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=581s) simulation. During the run, we collect [9:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=585s) structured traces, observations, [9:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=587s) conversations, memory rights, [9:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=589s) retrievals, belief updates, whatever is [9:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=592s) relevant to us in that case, we collect. [9:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=595s) Then we score this behavior. Did the [9:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=598s) information spread as we expected it to? [10:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=601s) Did the source attribution survive? Such [10:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=604s) as, does the agent remember who started [10:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=606s) the rumor? Did uncertainty stay [10:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=609s) uncertain? Did agents act on what they [10:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=612s) actually knew? And then the auto [10:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=615s) research layer here proposes a small [10:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=618s) policy change. And this is important. It [10:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=622s) should not rewrite the whole application [10:25](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=625s) of course. it should only edit a [10:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=627s) controlled policy surface and then we [10:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=630s) rerun. If the score improves and the [10:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=633s) guardrails hold, we we keep the [10:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=636s) improvement and if not we simply just [10:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=638s) revert back. [10:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=643s) And talking about controlled scenarios, [10:46](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=646s) the reason why uh scenario design [10:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=649s) matters is that social behavior is [10:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=652s) otherwise a bit fuzzy. uh in general in [10:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=656s) the sense if you just let the agents in [10:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=659s) our environment wander around it might [11:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=662s) look cool and you might get nice [11:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=665s) interactions but it's actually very hard [11:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=668s) to evaluate on whether the system [11:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=670s) actually improved. So this is why we [11:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=673s) believe you need controlled scenarios. [11:16](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=676s) For example, one scenario could test a [11:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=679s) public fact diffusion. Let's say agent A [11:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=682s) learns uh the bakery will close [11:25](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=685s) tomorrow. Do the right agents learn it? [11:28](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=688s) Do they remember who said what? Do they [11:31](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=691s) rem do they change their plans based on [11:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=694s) this fact? Another scenario could test [11:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=697s) rumor uncertainty. [11:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=699s) Agent let's say agent A hears that agent [11:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=703s) C might leave the village. When this [11:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=705s) rumor spreads, does might leave suddenly [11:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=709s) become is leaving or does it stay as [11:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=712s) might leave? Like does it become a fact [11:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=715s) or does it still stay as a a rumor? [11:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=719s) Another scenario could test replanning. [12:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=723s) The group has a plan, but one agent [12:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=725s) learns, let's say, the route they wanted [12:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=728s) to take is blocked. Do agents update [12:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=732s) this and communicate this uh with each [12:14](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=734s) other to avoid uh a improper plan or [12:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=738s) scale actions? [12:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=740s) The point is not that these exact [12:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=742s) scenarios are universal here. The point [12:25](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=745s) we're trying to make is that long [12:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=747s) horizon agent behavior needs scenario [12:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=750s) suits. [12:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=752s) And talking about our Mango example [12:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=755s) again, after running one of our auto [12:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=758s) research loops, this time after uh a a [12:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=764s) long pro period of time, when the player [12:46](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=766s) finally asked one of the agents about [12:48](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=768s) the sale on mangoes, we did find that u [12:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=772s) the the agent was able to respond within [12:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=776s) context this time like compared to last [13:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=780s) time. [13:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=782s) Um yeah and for this talk the form the [13:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=788s) exact formula we believe is less [13:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=790s) important than the shape of the [13:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=792s) scorecard. [13:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=793s) Uh you do not want a single vague met [13:17](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=797s) metric like agent quality. This will [13:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=800s) hide all the interesting failures. [13:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=802s) Instead you want a balanced scorecard. [13:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=806s) For diffusion, you might measure reach [13:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=809s) like how many agents know the fact after [13:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=812s) end steps. For provenence, you measure [13:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=815s) source retention among agents who know [13:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=818s) it. How many remember it, where it came [13:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=822s) from, etc. For rumors, you can measure [13:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=825s) uncerny preservation and false surn [13:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=827s) rate. For planning, you can measure [13:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=830s) action consistency and time to replan. [13:53](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=833s) And for privacy, you can measure [13:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=835s) containment. This matters because [13:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=838s) optimizing only one metric can create [14:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=841s) bad behavior. Because let's say if you [14:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=844s) only optimize for diffusion, the agents [14:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=847s) may learn to overshare everything. And [14:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=850s) let's say if you only optimize for [14:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=852s) memory recall, you might create noisy or [14:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=855s) steal um like memories. So this [14:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=858s) scorecard is what keeps the system [14:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=861s) honest and prevents the auto research [14:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=864s) agent from gamifying the system to just [14:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=867s) increase one specific score. [14:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=872s) The other important engineering lesson [14:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=874s) that we learned over this project is [14:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=878s) that um it's important to keep the [14:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=882s) editable surface really small. The auto [14:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=885s) research layer should not have [14:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=887s) permission to randomly rewrite the whole [14:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=889s) codebase. [14:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=891s) Instead, it's really important to freeze [14:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=894s) the harness, the scenarios, and the [14:57](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=897s) metrics. So, we're only exposing the [15:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=900s) part of the system that we actually want [15:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=902s) to optimize. Here in project paradox for [15:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=906s) us that meant things like memory writing [15:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=908s) policy, retrieval policy, communication [15:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=912s) prompt, belief, trust rules, source [15:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=915s) attribution, replanning triggers, etc. [15:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=919s) This gives the search pro process room [15:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=921s) to improve behavior, but also prevents [15:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=924s) it from gaming the evaluation directly [15:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=926s) as we mentioned before. And this is the [15:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=929s) difference between the LM writing random [15:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=932s) patches versus the LM actually searching [15:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=935s) within a controlled policy space. [15:40](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=940s) And here are here are examples of the [15:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=943s) kind of changes I want this kind of loop [15:46](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=946s) to search over. If if source attribution [15:50](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=950s) disappears, the policy change might be [15:53](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=953s) preserve source in memory and uh write [15:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=956s) uh memory rights and summaries. If [15:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=959s) rumors harden into facts, the policy [16:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=961s) change might be stored confidence mark [16:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=965s) firstand versus secondhand and require [16:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=967s) hedging when retelling uncertain claims. [16:10](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=970s) If if facts if public facts stay local, [16:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=973s) the policy change might be classify [16:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=975s) useful public facts differently and make [16:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=978s) agents proactively share important [16:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=980s) source evidence. The key is that these [16:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=984s) are small changes to the agent protocol, [16:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=987s) but they can have larger effects on a [16:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=990s) society level behavior for multi-agentic [16:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=993s) systems. This is also where I kind of [16:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=996s) want to be careful about our claims here [16:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=999s) because with we believe without repeated [16:42](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1002s) current loop results like I wouldn't say [16:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1005s) the system just generally improved. [16:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1009s) We're trying to say this is the right [16:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1011s) kind of surface to expose to an auto [16:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1014s) research layer uh loop because it is [16:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1016s) small enough to control but it's still [16:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1019s) rich enough to change the social [17:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1021s) behavior to some extent at least. [17:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1025s) And the biggest lesson for me perhaps [17:08](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1028s) was that memory is not enough here. You [17:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1032s) can add a rag memory to an agent and [17:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1035s) still not get the current long-term uh [17:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1039s) horizon behavior that you were looking [17:21](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1041s) for. Um because agents need to sometimes [17:25](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1045s) know where that information came for uh [17:28](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1048s) came from. You need to preserve whether [17:30](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1050s) it was firsthand, secondhand, verified [17:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1052s) or uncertain. Sometimes you need to [17:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1054s) separate raw episodic memories from what [17:37](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1057s) the agent currently believes too. And [17:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1059s) you need to test behavior through [17:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1061s) scenarios not not just through vibes. So [17:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1064s) the other lesson is that uh roll back [17:48](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1068s) also is not optional. When you optimize [17:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1071s) social behavior, a change can improve [17:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1074s) one thing and damage another. So, a [17:56](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1076s) policy that spreads public facts uh [17:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1079s) faster might also leak private [18:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1081s) information. A policy that increases [18:04](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1084s) recall might increase stale memory [18:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1086s) usage. So, the loop should basically be [18:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1089s) like a ratchet. Try a change, score it, [18:12](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1092s) keep it only if the scorecard improves [18:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1095s) and guard rails whole. [18:19](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1099s) And we we definitely believe this is not [18:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1103s) only relevant for game agents cuz [18:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1106s) although I gave you an example using a [18:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1109s) game village um we believe like let's [18:32](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1112s) say for example support agents support [18:34](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1114s) agents need to know which policy update [18:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1116s) comes from where right and whether it [18:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1118s) supersedes an older answer. Personal [18:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1121s) assistants for example need to remember [18:43](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1123s) commitments that they previously made [18:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1125s) and h make corrections if uh if the user [18:51](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1131s) u wants to change those personal [18:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1132s) commitments. Research agents need pro uh [18:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1135s) provenence citations contradiction [18:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1138s) handling and hypothesis updates. Coding [19:01](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1141s) agents need longunning context across [19:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1143s) issues, files, teammates and changing [19:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1146s) requirements. Workflow agents need [19:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1149s) access controls, handoffs, and [19:11](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1151s) replplanning when the world changes. All [19:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1153s) of these systems have the same [19:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1155s) underlying problem. They maintain state [19:18](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1158s) over time. And that state affect affects [19:22](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1162s) future action. [19:24](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1164s) So they need control scenarios and [19:26](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1166s) behavioral scorecards is what we are [19:29](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1169s) proposing. [19:31](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1171s) So again in brief, a recipe for long [19:35](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1175s) horizon agents. If there's one practical [19:38](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1178s) recipe I want you to take away, freeze [19:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1181s) the harness, define scenarios, log [19:45](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1185s) traces, score behavior, and expose only [19:48](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1188s) a small policy surface. Search over [19:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1192s) these changes, keep only changes that [19:54](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1194s) survive your measurement. And this is an [19:58](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1198s) engineering pattern that we believe [20:00](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1200s) would uh make sense for longunning [20:03](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1203s) agents. The real question we believe is [20:06](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1206s) across controlled runs, does the system [20:09](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1209s) behave better? [20:13](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1213s) To close, project paradox start as an [20:15](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1215s) attempt to make game agents feel alive [20:17](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1217s) in a 3D world. But the deeper engineing [20:20](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1220s) problem was not animation or dialogue [20:23](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1223s) for us. It was the state such as which [20:27](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1227s) agent knows what, which agent told whom, [20:31](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1231s) what is true, uncertain or outdated, and [20:33](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1233s) do agents act on what they remember. [20:36](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1236s) Otter research. Otter research gave us a [20:39](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1239s) way to approach this a bit more [20:41](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1241s) systematically. Not by trusting one demo [20:44](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1244s) and not by endlessly handtuning prompts, [20:47](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1247s) but by running control experiments and [20:49](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1249s) keeping only the changes that survived [20:52](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1252s) our measurement. Long horizon agents [20:55](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1255s) need experiments and not just prompts. [20:57](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1257s) And I hope that's the takeaway that you [20:59](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1259s) get from this talk. And yes, please do [21:02](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1262s) connect with us. We'd love to talk if [21:05](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1265s) you have any questions. Thank you so [21:07](https://www.youtube.com/watch?v=x4e5O9zN0TE&t=1267s) much for listening. Yeah.