[
  {
    "startMs": 12639,
    "text": "Hi everyone, I'm Arena, former engineer"
  },
  {
    "startMs": 15920,
    "text": "at Microsoft and Supercell. And today I"
  },
  {
    "startMs": 19039,
    "text": "want to talk about auto research in a"
  },
  {
    "startMs": 21359,
    "text": "multi- aent AI village. I will use a"
  },
  {
    "startMs": 25119,
    "text": "video game like AI Village as a running"
  },
  {
    "startMs": 27760,
    "text": "example here, but the broader question"
  },
  {
    "startMs": 30080,
    "text": "is one I think many AI engineers are"
  },
  {
    "startMs": 32960,
    "text": "starting to run into. How do we evaluate"
  },
  {
    "startMs": 36399,
    "text": "and improve agents that carry state over"
  },
  {
    "startMs": 39520,
    "text": "a long period of time?"
  },
  {
    "startMs": 43680,
    "text": "Before I get into the auto research"
  },
  {
    "startMs": 45920,
    "text": "layer, I want to talk a bit about"
  },
  {
    "startMs": 47840,
    "text": "project paradox."
  },
  {
    "startMs": 49920,
    "text": "We developed project paradox at"
  },
  {
    "startMs": 52239,
    "text": "supercell's AI innovation lab. Me and my"
  },
  {
    "startMs": 56079,
    "text": "teammate are natural money."
  },
  {
    "startMs": 59199,
    "text": "We built a modular AI framework that"
  },
  {
    "startMs": 62239,
    "text": "allows any developer to plug in"
  },
  {
    "startMs": 64640,
    "text": "intelligent autonomous agents within a"
  },
  {
    "startMs": 67360,
    "text": "video game that can interact, compete or"
  },
  {
    "startMs": 70640,
    "text": "cooperate with other players or agents"
  },
  {
    "startMs": 73119,
    "text": "as well and place them uh and make them"
  },
  {
    "startMs": 76080,
    "text": "into dynamic game companions."
  },
  {
    "startMs": 80479,
    "text": "Now to give examples of what these"
  },
  {
    "startMs": 82720,
    "text": "agents can do, the agents can move with"
  },
  {
    "startMs": 86159,
    "text": "intent. They can go to any location or"
  },
  {
    "startMs": 89680,
    "text": "person and they're guided by their own"
  },
  {
    "startMs": 92079,
    "text": "memories, emotion, or curiosity."
  },
  {
    "startMs": 95680,
    "text": "These agents can interact with the"
  },
  {
    "startMs": 97680,
    "text": "world. They can pick up objects, drop"
  },
  {
    "startMs": 100000,
    "text": "them anywhere, and they're also aware"
  },
  {
    "startMs": 102240,
    "text": "about the context in their own"
  },
  {
    "startMs": 104079,
    "text": "environment, such as the objects or"
  },
  {
    "startMs": 106479,
    "text": "other characters or agents as well. I"
  },
  {
    "startMs": 109600,
    "text": "would also like to note that game"
  },
  {
    "startMs": 111600,
    "text": "developers can also add new actions for"
  },
  {
    "startMs": 114799,
    "text": "these agents to accomplish within our"
  },
  {
    "startMs": 116960,
    "text": "framework as well. Instead of just"
  },
  {
    "startMs": 118799,
    "text": "dropping or uh placing objects,"
  },
  {
    "startMs": 124079,
    "text": "agents can also obviously react to"
  },
  {
    "startMs": 126560,
    "text": "what's happening around them. And these"
  },
  {
    "startMs": 130319,
    "text": "events that happen around them affect"
  },
  {
    "startMs": 132400,
    "text": "their own beliefs and emotions on the"
  },
  {
    "startMs": 134959,
    "text": "fly as well. And of course, it wouldn't"
  },
  {
    "startMs": 138160,
    "text": "be complete if agents can't start"
  },
  {
    "startMs": 140319,
    "text": "conversations, right? agents can in this"
  },
  {
    "startMs": 143520,
    "text": "scenario approach other agents or even"
  },
  {
    "startMs": 146959,
    "text": "the player as well and this makes the"
  },
  {
    "startMs": 150000,
    "text": "game feel more alive. And of course"
  },
  {
    "startMs": 152319,
    "text": "these conversations are stored within"
  },
  {
    "startMs": 154480,
    "text": "their memory and is according to their"
  },
  {
    "startMs": 157200,
    "text": "own um and affect their own emotions and"
  },
  {
    "startMs": 160560,
    "text": "beliefs or goals as well."
  },
  {
    "startMs": 163599,
    "text": "And al together these agents make our"
  },
  {
    "startMs": 166319,
    "text": "multi- aentic framework."
  },
  {
    "startMs": 171440,
    "text": "Um yeah"
  },
  {
    "startMs": 173760,
    "text": "yeah one second"
  },
  {
    "startMs": 177120,
    "text": "so the architecture was intentionally"
  },
  {
    "startMs": 180400,
    "text": "stateful behind this. The first"
  },
  {
    "startMs": 183200,
    "text": "important part was per agent memory."
  },
  {
    "startMs": 187040,
    "text": "Each agent has its own memory namespace"
  },
  {
    "startMs": 190640,
    "text": "backed by rag. So memory did not bleed"
  },
  {
    "startMs": 194080,
    "text": "between agents."
  },
  {
    "startMs": 196239,
    "text": "Second, we tracked emotion as a small"
  },
  {
    "startMs": 198800,
    "text": "vector. So after an event or"
  },
  {
    "startMs": 201519,
    "text": "conversation, the system could update"
  },
  {
    "startMs": 203440,
    "text": "values like joy, sadness, fear, anger,"
  },
  {
    "startMs": 207280,
    "text": "or disgust."
  },
  {
    "startMs": 209920,
    "text": "Third, agents had belief scores towards"
  },
  {
    "startMs": 213280,
    "text": "other agents and the player. You can"
  },
  {
    "startMs": 216400,
    "text": "think of this as a trust matrix."
  },
  {
    "startMs": 218319,
    "text": "Basically like after the interaction"
  },
  {
    "startMs": 221120,
    "text": "happens, the LM basically decides"
  },
  {
    "startMs": 223440,
    "text": "whether the trust score should go up,"
  },
  {
    "startMs": 225680,
    "text": "down, or whether it shouldn't change at"
  },
  {
    "startMs": 227680,
    "text": "all. And fourth, every memory receives"
  },
  {
    "startMs": 231760,
    "text": "an important score. Um to to explain"
  },
  {
    "startMs": 235280,
    "text": "this better, like let's say you had"
  },
  {
    "startMs": 237360,
    "text": "dinner a few days ago, you probably"
  },
  {
    "startMs": 238959,
    "text": "wouldn't remember what you had for"
  },
  {
    "startMs": 240480,
    "text": "dinner, right? But um"
  },
  {
    "startMs": 244080,
    "text": "if someone was murdered a few days ago,"
  },
  {
    "startMs": 246239,
    "text": "you definitely remember that. So the"
  },
  {
    "startMs": 248879,
    "text": "agent will evaluate or the LM will"
  },
  {
    "startMs": 251519,
    "text": "evaluate uh an important score of an"
  },
  {
    "startMs": 254080,
    "text": "event and if it crosses a threshold, it"
  },
  {
    "startMs": 256880,
    "text": "will store that specific memory uh in a"
  },
  {
    "startMs": 260880,
    "text": "separate cache so that important context"
  },
  {
    "startMs": 264000,
    "text": "can be retrieved better later on."
  },
  {
    "startMs": 268160,
    "text": "And here's an example of it just"
  },
  {
    "startMs": 271280,
    "text": "working. Um, we going to ask one of the"
  },
  {
    "startMs": 274479,
    "text": "characters to go on a picnic with us."
  },
  {
    "startMs": 276880,
    "text": "Here, uh, our character Blossom"
  },
  {
    "startMs": 280639,
    "text": "um, decides to pick up a pastry and go"
  },
  {
    "startMs": 283120,
    "text": "to the picnic area because we asked her"
  },
  {
    "startMs": 285360,
    "text": "to do so. Keep in mind during the"
  },
  {
    "startMs": 288639,
    "text": "conversation in the background, she"
  },
  {
    "startMs": 290880,
    "text": "plans all of these sequence of actions"
  },
  {
    "startMs": 292880,
    "text": "to accomplish. And one when we talk to"
  },
  {
    "startMs": 296240,
    "text": "her afterwards, she will also reply"
  },
  {
    "startMs": 298960,
    "text": "within context as well."
  },
  {
    "startMs": 303919,
    "text": "Yeah."
  },
  {
    "startMs": 306080,
    "text": "But this is where an interesting problem"
  },
  {
    "startMs": 309759,
    "text": "actually started. As you saw in the last"
  },
  {
    "startMs": 313039,
    "text": "example, like for shortterm game play,"
  },
  {
    "startMs": 316080,
    "text": "this our architecture worked pretty"
  },
  {
    "startMs": 318400,
    "text": "well. like a character could make a"
  },
  {
    "startMs": 320880,
    "text": "plan, move around, talk and remember the"
  },
  {
    "startMs": 323759,
    "text": "recent interaction and respond to us or"
  },
  {
    "startMs": 326800,
    "text": "other characters as well. But over"
  },
  {
    "startMs": 329759,
    "text": "longer horizons,"
  },
  {
    "startMs": 332000,
    "text": "this is where we notice the social"
  },
  {
    "startMs": 333680,
    "text": "consistency start to get weaker. So in"
  },
  {
    "startMs": 336639,
    "text": "this example, we have one agent"
  },
  {
    "startMs": 338800,
    "text": "spreading a rumor about a sale on"
  },
  {
    "startMs": 341759,
    "text": "mangoes to another agent and that agent"
  },
  {
    "startMs": 344960,
    "text": "receives that information and goes and"
  },
  {
    "startMs": 347199,
    "text": "tells another agent about it. Later on,"
  },
  {
    "startMs": 350639,
    "text": "after a number of events that occurred"
  },
  {
    "startMs": 353360,
    "text": "in between, when the player asks one of"
  },
  {
    "startMs": 355919,
    "text": "the agents about the mangoes, it doesn't"
  },
  {
    "startMs": 358880,
    "text": "exactly store that context that we were"
  },
  {
    "startMs": 361759,
    "text": "expecting or it doesn't give us the"
  },
  {
    "startMs": 363199,
    "text": "context that we kind of wanted to. And"
  },
  {
    "startMs": 366720,
    "text": "this is where things are starting to get"
  },
  {
    "startMs": 368960,
    "text": "messy naturally. Like the system may"
  },
  {
    "startMs": 372400,
    "text": "remember the rough topic but lose the"
  },
  {
    "startMs": 375520,
    "text": "source of the topic. A rumor may become"
  },
  {
    "startMs": 378080,
    "text": "certain instead of just a rumor like the"
  },
  {
    "startMs": 381360,
    "text": "agent might state it as a fact or um an"
  },
  {
    "startMs": 384960,
    "text": "agent might know a fact but fail to"
  },
  {
    "startMs": 387360,
    "text": "execute fail to remember it while"
  },
  {
    "startMs": 389199,
    "text": "creating a plan for its actions. So the"
  },
  {
    "startMs": 393120,
    "text": "question here became how do we improve a"
  },
  {
    "startMs": 396080,
    "text": "multi-agentic system over longunning"
  },
  {
    "startMs": 398960,
    "text": "social behavior and not just over one"
  },
  {
    "startMs": 401759,
    "text": "response."
  },
  {
    "startMs": 404000,
    "text": "And this is where we wanted to bring in"
  },
  {
    "startMs": 407280,
    "text": "auto research. As you all know, uh few"
  },
  {
    "startMs": 410560,
    "text": "months ago, Karpathi posted out auto"
  },
  {
    "startMs": 413840,
    "text": "research and this this made us"
  },
  {
    "startMs": 416479,
    "text": "immediately very curious. Uh perhaps we"
  },
  {
    "startMs": 420319,
    "text": "can make the system run experiments uh"
  },
  {
    "startMs": 423360,
    "text": "on itself and can we use this for our"
  },
  {
    "startMs": 425919,
    "text": "system as well. So what we understood is"
  },
  {
    "startMs": 429840,
    "text": "instead of manually tuning a prompt or"
  },
  {
    "startMs": 432240,
    "text": "watching one nice demo, we could define"
  },
  {
    "startMs": 435840,
    "text": "sen a scenario suit, run the agents,"
  },
  {
    "startMs": 439039,
    "text": "collect traces, score the behavior and"
  },
  {
    "startMs": 442080,
    "text": "change a small policy surface and only"
  },
  {
    "startMs": 444880,
    "text": "keep the changes that actually improve"
  },
  {
    "startMs": 447280,
    "text": "the score. And this is where we're"
  },
  {
    "startMs": 449520,
    "text": "trying to bridge project paradox with"
  },
  {
    "startMs": 452240,
    "text": "auto research. So at this point"
  },
  {
    "startMs": 454960,
    "text": "basically our multi-agentic framework"
  },
  {
    "startMs": 457360,
    "text": "project paradox is more like a lab bench"
  },
  {
    "startMs": 460560,
    "text": "and auto research becomes the"
  },
  {
    "startMs": 462400,
    "text": "experimental loop around it. And"
  },
  {
    "startMs": 465599,
    "text": "importantly this is not only about"
  },
  {
    "startMs": 468240,
    "text": "improving rag retrieval. The broader"
  },
  {
    "startMs": 470960,
    "text": "framing is optimizing the agent protocol"
  },
  {
    "startMs": 474240,
    "text": "like how do agents write memories,"
  },
  {
    "startMs": 476639,
    "text": "retrieve them, communicate uncertainty,"
  },
  {
    "startMs": 479840,
    "text": "update trust attribute sources and"
  },
  {
    "startMs": 482000,
    "text": "replan around new facts. basically"
  },
  {
    "startMs": 488319,
    "text": "um yeah in this context uh"
  },
  {
    "startMs": 495360,
    "text": "oh yeah in this context art research is"
  },
  {
    "startMs": 498479,
    "text": "not another agent in the village like I"
  },
  {
    "startMs": 501039,
    "text": "said it's a meta system outside the"
  },
  {
    "startMs": 503759,
    "text": "village the villagers have local"
  },
  {
    "startMs": 506479,
    "text": "perspectives of course they only know"
  },
  {
    "startMs": 510240,
    "text": "what they saw heard remembered or"
  },
  {
    "startMs": 512719,
    "text": "inferred because there isn't a common"
  },
  {
    "startMs": 514800,
    "text": "memory database in between them."
  },
  {
    "startMs": 517440,
    "text": "Information only travels once uh other"
  },
  {
    "startMs": 520719,
    "text": "agents communicate them properly."
  },
  {
    "startMs": 523839,
    "text": "The auto research layer has a different"
  },
  {
    "startMs": 525920,
    "text": "job here. It reads the full traces of a"
  },
  {
    "startMs": 529040,
    "text": "run, compares what happens against the"
  },
  {
    "startMs": 532000,
    "text": "scenario ground truth,"
  },
  {
    "startMs": 534560,
    "text": "uh scores the behavior and proposes a"
  },
  {
    "startMs": 537440,
    "text": "constrained"
  },
  {
    "startMs": 539600,
    "text": "change to the agent protocol or"
  },
  {
    "startMs": 541760,
    "text": "cognitive policy. Then it reruns the"
  },
  {
    "startMs": 544800,
    "text": "scenario and asks society level behavior"
  },
  {
    "startMs": 547600,
    "text": "like did society level behavior get"
  },
  {
    "startMs": 549680,
    "text": "better. This is the key shift we were"
  },
  {
    "startMs": 552720,
    "text": "trying to look for. So we were no longer"
  },
  {
    "startMs": 555839,
    "text": "evaluating one answer. We were"
  },
  {
    "startMs": 557839,
    "text": "evaluating an entire run."
  },
  {
    "startMs": 561839,
    "text": "And this is what one of the loops would"
  },
  {
    "startMs": 564480,
    "text": "look like. Like first we define a"
  },
  {
    "startMs": 567519,
    "text": "control scenario which I'll elaborate a"
  },
  {
    "startMs": 569760,
    "text": "bit more about later. For example, one"
  },
  {
    "startMs": 572959,
    "text": "agent learns a public fact or one agent"
  },
  {
    "startMs": 576160,
    "text": "hears a rumor. Uh that could be a"
  },
  {
    "startMs": 578959,
    "text": "controlled scenario. Then we run the"
  },
  {
    "startMs": 581600,
    "text": "simulation. During the run, we collect"
  },
  {
    "startMs": 585040,
    "text": "structured traces, observations,"
  },
  {
    "startMs": 587839,
    "text": "conversations, memory rights,"
  },
  {
    "startMs": 589440,
    "text": "retrievals, belief updates, whatever is"
  },
  {
    "startMs": 592160,
    "text": "relevant to us in that case, we collect."
  },
  {
    "startMs": 595680,
    "text": "Then we score this behavior. Did the"
  },
  {
    "startMs": 598959,
    "text": "information spread as we expected it to?"
  },
  {
    "startMs": 601760,
    "text": "Did the source attribution survive? Such"
  },
  {
    "startMs": 604160,
    "text": "as, does the agent remember who started"
  },
  {
    "startMs": 606880,
    "text": "the rumor? Did uncertainty stay"
  },
  {
    "startMs": 609440,
    "text": "uncertain? Did agents act on what they"
  },
  {
    "startMs": 612720,
    "text": "actually knew? And then the auto"
  },
  {
    "startMs": 615920,
    "text": "research layer here proposes a small"
  },
  {
    "startMs": 618560,
    "text": "policy change. And this is important. It"
  },
  {
    "startMs": 622399,
    "text": "should not rewrite the whole application"
  },
  {
    "startMs": 625040,
    "text": "of course. it should only edit a"
  },
  {
    "startMs": 627760,
    "text": "controlled policy surface and then we"
  },
  {
    "startMs": 630959,
    "text": "rerun. If the score improves and the"
  },
  {
    "startMs": 633600,
    "text": "guardrails hold, we we keep the"
  },
  {
    "startMs": 636640,
    "text": "improvement and if not we simply just"
  },
  {
    "startMs": 638959,
    "text": "revert back."
  },
  {
    "startMs": 643200,
    "text": "And talking about controlled scenarios,"
  },
  {
    "startMs": 646240,
    "text": "the reason why uh scenario design"
  },
  {
    "startMs": 649839,
    "text": "matters is that social behavior is"
  },
  {
    "startMs": 652800,
    "text": "otherwise a bit fuzzy. uh in general in"
  },
  {
    "startMs": 656399,
    "text": "the sense if you just let the agents in"
  },
  {
    "startMs": 659680,
    "text": "our environment wander around it might"
  },
  {
    "startMs": 662880,
    "text": "look cool and you might get nice"
  },
  {
    "startMs": 665040,
    "text": "interactions but it's actually very hard"
  },
  {
    "startMs": 668000,
    "text": "to evaluate on whether the system"
  },
  {
    "startMs": 670240,
    "text": "actually improved. So this is why we"
  },
  {
    "startMs": 673360,
    "text": "believe you need controlled scenarios."
  },
  {
    "startMs": 676720,
    "text": "For example, one scenario could test a"
  },
  {
    "startMs": 679279,
    "text": "public fact diffusion. Let's say agent A"
  },
  {
    "startMs": 682560,
    "text": "learns uh the bakery will close"
  },
  {
    "startMs": 685360,
    "text": "tomorrow. Do the right agents learn it?"
  },
  {
    "startMs": 688480,
    "text": "Do they remember who said what? Do they"
  },
  {
    "startMs": 691519,
    "text": "rem do they change their plans based on"
  },
  {
    "startMs": 694240,
    "text": "this fact? Another scenario could test"
  },
  {
    "startMs": 697600,
    "text": "rumor uncertainty."
  },
  {
    "startMs": 699680,
    "text": "Agent let's say agent A hears that agent"
  },
  {
    "startMs": 703360,
    "text": "C might leave the village. When this"
  },
  {
    "startMs": 705920,
    "text": "rumor spreads, does might leave suddenly"
  },
  {
    "startMs": 709600,
    "text": "become is leaving or does it stay as"
  },
  {
    "startMs": 712640,
    "text": "might leave? Like does it become a fact"
  },
  {
    "startMs": 715600,
    "text": "or does it still stay as a a rumor?"
  },
  {
    "startMs": 719839,
    "text": "Another scenario could test replanning."
  },
  {
    "startMs": 723279,
    "text": "The group has a plan, but one agent"
  },
  {
    "startMs": 725839,
    "text": "learns, let's say, the route they wanted"
  },
  {
    "startMs": 728800,
    "text": "to take is blocked. Do agents update"
  },
  {
    "startMs": 732399,
    "text": "this and communicate this uh with each"
  },
  {
    "startMs": 734880,
    "text": "other to avoid uh a improper plan or"
  },
  {
    "startMs": 738000,
    "text": "scale actions?"
  },
  {
    "startMs": 740079,
    "text": "The point is not that these exact"
  },
  {
    "startMs": 742320,
    "text": "scenarios are universal here. The point"
  },
  {
    "startMs": 745440,
    "text": "we're trying to make is that long"
  },
  {
    "startMs": 747360,
    "text": "horizon agent behavior needs scenario"
  },
  {
    "startMs": 750079,
    "text": "suits."
  },
  {
    "startMs": 752720,
    "text": "And talking about our Mango example"
  },
  {
    "startMs": 755600,
    "text": "again, after running one of our auto"
  },
  {
    "startMs": 758480,
    "text": "research loops, this time after uh a a"
  },
  {
    "startMs": 764000,
    "text": "long pro period of time, when the player"
  },
  {
    "startMs": 766480,
    "text": "finally asked one of the agents about"
  },
  {
    "startMs": 768240,
    "text": "the sale on mangoes, we did find that u"
  },
  {
    "startMs": 772639,
    "text": "the the agent was able to respond within"
  },
  {
    "startMs": 776320,
    "text": "context this time like compared to last"
  },
  {
    "startMs": 780000,
    "text": "time."
  },
  {
    "startMs": 782880,
    "text": "Um yeah and for this talk the form the"
  },
  {
    "startMs": 788000,
    "text": "exact formula we believe is less"
  },
  {
    "startMs": 790800,
    "text": "important than the shape of the"
  },
  {
    "startMs": 792320,
    "text": "scorecard."
  },
  {
    "startMs": 793920,
    "text": "Uh you do not want a single vague met"
  },
  {
    "startMs": 797519,
    "text": "metric like agent quality. This will"
  },
  {
    "startMs": 800639,
    "text": "hide all the interesting failures."
  },
  {
    "startMs": 802959,
    "text": "Instead you want a balanced scorecard."
  },
  {
    "startMs": 806399,
    "text": "For diffusion, you might measure reach"
  },
  {
    "startMs": 809760,
    "text": "like how many agents know the fact after"
  },
  {
    "startMs": 812240,
    "text": "end steps. For provenence, you measure"
  },
  {
    "startMs": 815920,
    "text": "source retention among agents who know"
  },
  {
    "startMs": 818639,
    "text": "it. How many remember it, where it came"
  },
  {
    "startMs": 822320,
    "text": "from, etc. For rumors, you can measure"
  },
  {
    "startMs": 825120,
    "text": "uncerny preservation and false surn"
  },
  {
    "startMs": 827760,
    "text": "rate. For planning, you can measure"
  },
  {
    "startMs": 830399,
    "text": "action consistency and time to replan."
  },
  {
    "startMs": 833600,
    "text": "And for privacy, you can measure"
  },
  {
    "startMs": 835760,
    "text": "containment. This matters because"
  },
  {
    "startMs": 838240,
    "text": "optimizing only one metric can create"
  },
  {
    "startMs": 841440,
    "text": "bad behavior. Because let's say if you"
  },
  {
    "startMs": 844320,
    "text": "only optimize for diffusion, the agents"
  },
  {
    "startMs": 847279,
    "text": "may learn to overshare everything. And"
  },
  {
    "startMs": 850320,
    "text": "let's say if you only optimize for"
  },
  {
    "startMs": 852240,
    "text": "memory recall, you might create noisy or"
  },
  {
    "startMs": 855120,
    "text": "steal um like memories. So this"
  },
  {
    "startMs": 858880,
    "text": "scorecard is what keeps the system"
  },
  {
    "startMs": 861040,
    "text": "honest and prevents the auto research"
  },
  {
    "startMs": 864560,
    "text": "agent from gamifying the system to just"
  },
  {
    "startMs": 867680,
    "text": "increase one specific score."
  },
  {
    "startMs": 872399,
    "text": "The other important engineering lesson"
  },
  {
    "startMs": 874880,
    "text": "that we learned over this project is"
  },
  {
    "startMs": 878160,
    "text": "that um it's important to keep the"
  },
  {
    "startMs": 882160,
    "text": "editable surface really small. The auto"
  },
  {
    "startMs": 885440,
    "text": "research layer should not have"
  },
  {
    "startMs": 887519,
    "text": "permission to randomly rewrite the whole"
  },
  {
    "startMs": 889920,
    "text": "codebase."
  },
  {
    "startMs": 891440,
    "text": "Instead, it's really important to freeze"
  },
  {
    "startMs": 894560,
    "text": "the harness, the scenarios, and the"
  },
  {
    "startMs": 897440,
    "text": "metrics. So, we're only exposing the"
  },
  {
    "startMs": 900399,
    "text": "part of the system that we actually want"
  },
  {
    "startMs": 902720,
    "text": "to optimize. Here in project paradox for"
  },
  {
    "startMs": 906399,
    "text": "us that meant things like memory writing"
  },
  {
    "startMs": 908959,
    "text": "policy, retrieval policy, communication"
  },
  {
    "startMs": 912320,
    "text": "prompt, belief, trust rules, source"
  },
  {
    "startMs": 915519,
    "text": "attribution, replanning triggers, etc."
  },
  {
    "startMs": 919040,
    "text": "This gives the search pro process room"
  },
  {
    "startMs": 921920,
    "text": "to improve behavior, but also prevents"
  },
  {
    "startMs": 924079,
    "text": "it from gaming the evaluation directly"
  },
  {
    "startMs": 926639,
    "text": "as we mentioned before. And this is the"
  },
  {
    "startMs": 929360,
    "text": "difference between the LM writing random"
  },
  {
    "startMs": 932160,
    "text": "patches versus the LM actually searching"
  },
  {
    "startMs": 935839,
    "text": "within a controlled policy space."
  },
  {
    "startMs": 940880,
    "text": "And here are here are examples of the"
  },
  {
    "startMs": 943760,
    "text": "kind of changes I want this kind of loop"
  },
  {
    "startMs": 946320,
    "text": "to search over. If if source attribution"
  },
  {
    "startMs": 950320,
    "text": "disappears, the policy change might be"
  },
  {
    "startMs": 953279,
    "text": "preserve source in memory and uh write"
  },
  {
    "startMs": 956880,
    "text": "uh memory rights and summaries. If"
  },
  {
    "startMs": 959360,
    "text": "rumors harden into facts, the policy"
  },
  {
    "startMs": 961839,
    "text": "change might be stored confidence mark"
  },
  {
    "startMs": 965360,
    "text": "firstand versus secondhand and require"
  },
  {
    "startMs": 967440,
    "text": "hedging when retelling uncertain claims."
  },
  {
    "startMs": 970399,
    "text": "If if facts if public facts stay local,"
  },
  {
    "startMs": 973519,
    "text": "the policy change might be classify"
  },
  {
    "startMs": 975759,
    "text": "useful public facts differently and make"
  },
  {
    "startMs": 978399,
    "text": "agents proactively share important"
  },
  {
    "startMs": 980560,
    "text": "source evidence. The key is that these"
  },
  {
    "startMs": 984720,
    "text": "are small changes to the agent protocol,"
  },
  {
    "startMs": 987759,
    "text": "but they can have larger effects on a"
  },
  {
    "startMs": 990320,
    "text": "society level behavior for multi-agentic"
  },
  {
    "startMs": 993199,
    "text": "systems. This is also where I kind of"
  },
  {
    "startMs": 996560,
    "text": "want to be careful about our claims here"
  },
  {
    "startMs": 999040,
    "text": "because with we believe without repeated"
  },
  {
    "startMs": 1002639,
    "text": "current loop results like I wouldn't say"
  },
  {
    "startMs": 1005120,
    "text": "the system just generally improved."
  },
  {
    "startMs": 1009199,
    "text": "We're trying to say this is the right"
  },
  {
    "startMs": 1011360,
    "text": "kind of surface to expose to an auto"
  },
  {
    "startMs": 1014240,
    "text": "research layer uh loop because it is"
  },
  {
    "startMs": 1016800,
    "text": "small enough to control but it's still"
  },
  {
    "startMs": 1019519,
    "text": "rich enough to change the social"
  },
  {
    "startMs": 1021120,
    "text": "behavior to some extent at least."
  },
  {
    "startMs": 1025199,
    "text": "And the biggest lesson for me perhaps"
  },
  {
    "startMs": 1028720,
    "text": "was that memory is not enough here. You"
  },
  {
    "startMs": 1032720,
    "text": "can add a rag memory to an agent and"
  },
  {
    "startMs": 1035760,
    "text": "still not get the current long-term uh"
  },
  {
    "startMs": 1039120,
    "text": "horizon behavior that you were looking"
  },
  {
    "startMs": 1041120,
    "text": "for. Um because agents need to sometimes"
  },
  {
    "startMs": 1045120,
    "text": "know where that information came for uh"
  },
  {
    "startMs": 1048160,
    "text": "came from. You need to preserve whether"
  },
  {
    "startMs": 1050240,
    "text": "it was firsthand, secondhand, verified"
  },
  {
    "startMs": 1052640,
    "text": "or uncertain. Sometimes you need to"
  },
  {
    "startMs": 1054960,
    "text": "separate raw episodic memories from what"
  },
  {
    "startMs": 1057200,
    "text": "the agent currently believes too. And"
  },
  {
    "startMs": 1059440,
    "text": "you need to test behavior through"
  },
  {
    "startMs": 1061360,
    "text": "scenarios not not just through vibes. So"
  },
  {
    "startMs": 1064960,
    "text": "the other lesson is that uh roll back"
  },
  {
    "startMs": 1068400,
    "text": "also is not optional. When you optimize"
  },
  {
    "startMs": 1071440,
    "text": "social behavior, a change can improve"
  },
  {
    "startMs": 1074320,
    "text": "one thing and damage another. So, a"
  },
  {
    "startMs": 1076799,
    "text": "policy that spreads public facts uh"
  },
  {
    "startMs": 1079520,
    "text": "faster might also leak private"
  },
  {
    "startMs": 1081679,
    "text": "information. A policy that increases"
  },
  {
    "startMs": 1084000,
    "text": "recall might increase stale memory"
  },
  {
    "startMs": 1086160,
    "text": "usage. So, the loop should basically be"
  },
  {
    "startMs": 1089360,
    "text": "like a ratchet. Try a change, score it,"
  },
  {
    "startMs": 1092799,
    "text": "keep it only if the scorecard improves"
  },
  {
    "startMs": 1095600,
    "text": "and guard rails whole."
  },
  {
    "startMs": 1099840,
    "text": "And we we definitely believe this is not"
  },
  {
    "startMs": 1103039,
    "text": "only relevant for game agents cuz"
  },
  {
    "startMs": 1106559,
    "text": "although I gave you an example using a"
  },
  {
    "startMs": 1109039,
    "text": "game village um we believe like let's"
  },
  {
    "startMs": 1112720,
    "text": "say for example support agents support"
  },
  {
    "startMs": 1114640,
    "text": "agents need to know which policy update"
  },
  {
    "startMs": 1116880,
    "text": "comes from where right and whether it"
  },
  {
    "startMs": 1118720,
    "text": "supersedes an older answer. Personal"
  },
  {
    "startMs": 1121280,
    "text": "assistants for example need to remember"
  },
  {
    "startMs": 1123760,
    "text": "commitments that they previously made"
  },
  {
    "startMs": 1125840,
    "text": "and h make corrections if uh if the user"
  },
  {
    "startMs": 1131039,
    "text": "u wants to change those personal"
  },
  {
    "startMs": 1132720,
    "text": "commitments. Research agents need pro uh"
  },
  {
    "startMs": 1135919,
    "text": "provenence citations contradiction"
  },
  {
    "startMs": 1138240,
    "text": "handling and hypothesis updates. Coding"
  },
  {
    "startMs": 1141440,
    "text": "agents need longunning context across"
  },
  {
    "startMs": 1143919,
    "text": "issues, files, teammates and changing"
  },
  {
    "startMs": 1146720,
    "text": "requirements. Workflow agents need"
  },
  {
    "startMs": 1149200,
    "text": "access controls, handoffs, and"
  },
  {
    "startMs": 1151120,
    "text": "replplanning when the world changes. All"
  },
  {
    "startMs": 1153440,
    "text": "of these systems have the same"
  },
  {
    "startMs": 1155600,
    "text": "underlying problem. They maintain state"
  },
  {
    "startMs": 1158640,
    "text": "over time. And that state affect affects"
  },
  {
    "startMs": 1162240,
    "text": "future action."
  },
  {
    "startMs": 1164480,
    "text": "So they need control scenarios and"
  },
  {
    "startMs": 1166880,
    "text": "behavioral scorecards is what we are"
  },
  {
    "startMs": 1169280,
    "text": "proposing."
  },
  {
    "startMs": 1171520,
    "text": "So again in brief, a recipe for long"
  },
  {
    "startMs": 1175440,
    "text": "horizon agents. If there's one practical"
  },
  {
    "startMs": 1178400,
    "text": "recipe I want you to take away, freeze"
  },
  {
    "startMs": 1181679,
    "text": "the harness, define scenarios, log"
  },
  {
    "startMs": 1185200,
    "text": "traces, score behavior, and expose only"
  },
  {
    "startMs": 1188480,
    "text": "a small policy surface. Search over"
  },
  {
    "startMs": 1192080,
    "text": "these changes, keep only changes that"
  },
  {
    "startMs": 1194799,
    "text": "survive your measurement. And this is an"
  },
  {
    "startMs": 1198400,
    "text": "engineering pattern that we believe"
  },
  {
    "startMs": 1200480,
    "text": "would uh make sense for longunning"
  },
  {
    "startMs": 1203360,
    "text": "agents. The real question we believe is"
  },
  {
    "startMs": 1206720,
    "text": "across controlled runs, does the system"
  },
  {
    "startMs": 1209360,
    "text": "behave better?"
  },
  {
    "startMs": 1213039,
    "text": "To close, project paradox start as an"
  },
  {
    "startMs": 1215520,
    "text": "attempt to make game agents feel alive"
  },
  {
    "startMs": 1217760,
    "text": "in a 3D world. But the deeper engineing"
  },
  {
    "startMs": 1220880,
    "text": "problem was not animation or dialogue"
  },
  {
    "startMs": 1223200,
    "text": "for us. It was the state such as which"
  },
  {
    "startMs": 1227280,
    "text": "agent knows what, which agent told whom,"
  },
  {
    "startMs": 1231200,
    "text": "what is true, uncertain or outdated, and"
  },
  {
    "startMs": 1233600,
    "text": "do agents act on what they remember."
  },
  {
    "startMs": 1236559,
    "text": "Otter research. Otter research gave us a"
  },
  {
    "startMs": 1239360,
    "text": "way to approach this a bit more"
  },
  {
    "startMs": 1241200,
    "text": "systematically. Not by trusting one demo"
  },
  {
    "startMs": 1244240,
    "text": "and not by endlessly handtuning prompts,"
  },
  {
    "startMs": 1247520,
    "text": "but by running control experiments and"
  },
  {
    "startMs": 1249840,
    "text": "keeping only the changes that survived"
  },
  {
    "startMs": 1252000,
    "text": "our measurement. Long horizon agents"
  },
  {
    "startMs": 1255120,
    "text": "need experiments and not just prompts."
  },
  {
    "startMs": 1257280,
    "text": "And I hope that's the takeaway that you"
  },
  {
    "startMs": 1259360,
    "text": "get from this talk. And yes, please do"
  },
  {
    "startMs": 1262880,
    "text": "connect with us. We'd love to talk if"
  },
  {
    "startMs": 1265200,
    "text": "you have any questions. Thank you so"
  },
  {
    "startMs": 1267120,
    "text": "much for listening. Yeah."
  }
]