[
  {
    "startMs": 12520,
    "text": "Um my name is Siraj Gupta. I lead"
  },
  {
    "startMs": 14400,
    "text": "Harness development at Warp. Um in case"
  },
  {
    "startMs": 16320,
    "text": "you haven't heard of Warp before, um we"
  },
  {
    "startMs": 18800,
    "text": "started off by building a"
  },
  {
    "startMs": 20400,
    "text": "state-of-the-art agentic development"
  },
  {
    "startMs": 22240,
    "text": "environment. Um we have about a million"
  },
  {
    "startMs": 24600,
    "text": "active users that use that. Um it was"
  },
  {
    "startMs": 26720,
    "text": "born out of a modern-day terminal."
  },
  {
    "startMs": 28720,
    "text": "Um and nowadays we're building a um"
  },
  {
    "startMs": 31160,
    "text": "cloud agent platform to enable teams to"
  },
  {
    "startMs": 33480,
    "text": "build software factories."
  },
  {
    "startMs": 35360,
    "text": "Um today I'm going to talk about um"
  },
  {
    "startMs": 38840,
    "text": "self-improving software factories, which"
  },
  {
    "startMs": 40880,
    "text": "I know might sound kind of buzzy, um"
  },
  {
    "startMs": 43440,
    "text": "but I'm going to break it down and"
  },
  {
    "startMs": 44440,
    "text": "hopefully you can walk out of here with"
  },
  {
    "startMs": 45400,
    "text": "something concrete that you can"
  },
  {
    "startMs": 46760,
    "text": "implement yourselves."
  },
  {
    "startMs": 48480,
    "text": "Um you've probably heard of both of"
  },
  {
    "startMs": 50040,
    "text": "these terms, self-improvement and"
  },
  {
    "startMs": 52200,
    "text": "software factories, used throughout this"
  },
  {
    "startMs": 53800,
    "text": "conference, but I'm sure you've heard of"
  },
  {
    "startMs": 55680,
    "text": "them independently."
  },
  {
    "startMs": 57240,
    "text": "Um"
  },
  {
    "startMs": 57880,
    "text": "you've heard about how"
  },
  {
    "startMs": 59720,
    "text": "self-improvement is about agents being"
  },
  {
    "startMs": 61320,
    "text": "able to"
  },
  {
    "startMs": 62440,
    "text": "um"
  },
  {
    "startMs": 63120,
    "text": "improve over time and agents and models"
  },
  {
    "startMs": 66120,
    "text": "um"
  },
  {
    "startMs": 66720,
    "text": "and, you know, phasing out humans from"
  },
  {
    "startMs": 68840,
    "text": "that loop, so that way the the"
  },
  {
    "startMs": 70920,
    "text": "improvement loop can be automatic. Um"
  },
  {
    "startMs": 73040,
    "text": "and I'm sure you've heard about software"
  },
  {
    "startMs": 74240,
    "text": "factories being"
  },
  {
    "startMs": 76000,
    "text": "um the way that like the software"
  },
  {
    "startMs": 78400,
    "text": "development paradigm is shifting from"
  },
  {
    "startMs": 80240,
    "text": "individual coding agents to building out"
  },
  {
    "startMs": 82680,
    "text": "automations that take work from triage"
  },
  {
    "startMs": 85320,
    "text": "up until production. Um but less has"
  },
  {
    "startMs": 87840,
    "text": "been said about how software factories"
  },
  {
    "startMs": 90400,
    "text": "themselves can self-improve over time,"
  },
  {
    "startMs": 92560,
    "text": "so that the the factory can get better,"
  },
  {
    "startMs": 94600,
    "text": "more efficient, faster. Um and I think"
  },
  {
    "startMs": 97560,
    "text": "this is going to become an extremely"
  },
  {
    "startMs": 98880,
    "text": "important part of our jobs as software"
  },
  {
    "startMs": 100920,
    "text": "engineers as we transition from building"
  },
  {
    "startMs": 103200,
    "text": "products to building factories and and"
  },
  {
    "startMs": 105360,
    "text": "maintaining them."
  },
  {
    "startMs": 106880,
    "text": "Um so let's dive in. Um"
  },
  {
    "startMs": 109760,
    "text": "self-improvement is a pretty broad"
  },
  {
    "startMs": 110880,
    "text": "umbrella, but I'm going to focus on"
  },
  {
    "startMs": 112880,
    "text": "three concrete ways that agents can"
  },
  {
    "startMs": 114960,
    "text": "improve over time, agent factories can"
  },
  {
    "startMs": 117040,
    "text": "improve over time. I'm going to start"
  },
  {
    "startMs": 118600,
    "text": "with um skills, then we'll talk about"
  },
  {
    "startMs": 120440,
    "text": "persistent memory, and then I'll finish"
  },
  {
    "startMs": 122160,
    "text": "off by talking about model routing."
  },
  {
    "startMs": 124479,
    "text": "Um so"
  },
  {
    "startMs": 126400,
    "text": "skills are a great way to give your"
  },
  {
    "startMs": 127920,
    "text": "agents procedural memory. I don't think"
  },
  {
    "startMs": 129399,
    "text": "I have to kind of spell that out for"
  },
  {
    "startMs": 130840,
    "text": "you. You describe a a procedure to your"
  },
  {
    "startMs": 133360,
    "text": "agent that you expect it to have to do a"
  },
  {
    "startMs": 135560,
    "text": "task over and over. Um for example,"
  },
  {
    "startMs": 138120,
    "text": "let's say you're building a triage"
  },
  {
    "startMs": 139400,
    "text": "agent, which is a pretty common thing"
  },
  {
    "startMs": 140640,
    "text": "you might build in a software factory."
  },
  {
    "startMs": 143040,
    "text": "Um"
  },
  {
    "startMs": 144480,
    "text": "and you give it a skill on how you want"
  },
  {
    "startMs": 146160,
    "text": "it to go about reproducing issues. Um"
  },
  {
    "startMs": 148400,
    "text": "those skills get stale over time. Um"
  },
  {
    "startMs": 150720,
    "text": "humans give feedback to the agents."
  },
  {
    "startMs": 152760,
    "text": "Agents learn things in their own um runs"
  },
  {
    "startMs": 155480,
    "text": "and trajectories."
  },
  {
    "startMs": 156840,
    "text": "Um and"
  },
  {
    "startMs": 158040,
    "text": "the like the skill that you previously"
  },
  {
    "startMs": 160040,
    "text": "gave it becomes dated. So, how do we fix"
  },
  {
    "startMs": 162440,
    "text": "this and let the agent improve over time"
  },
  {
    "startMs": 164360,
    "text": "using that skill?"
  },
  {
    "startMs": 165880,
    "text": "Well, one way to do it is to deploy an"
  },
  {
    "startMs": 167680,
    "text": "outer loop agent. Um and this is what"
  },
  {
    "startMs": 169959,
    "text": "we've done in our own internal software"
  },
  {
    "startMs": 171480,
    "text": "factory that's been really productive."
  },
  {
    "startMs": 173720,
    "text": "The idea is that your um"
  },
  {
    "startMs": 176760,
    "text": "your inner loop agent is the thing"
  },
  {
    "startMs": 178400,
    "text": "that's actually applying the skill. It's"
  },
  {
    "startMs": 180080,
    "text": "like doing the triage. And then you have"
  },
  {
    "startMs": 182000,
    "text": "this other outer loop agent that is"
  },
  {
    "startMs": 183560,
    "text": "observing your inner loop agent's runs"
  },
  {
    "startMs": 186200,
    "text": "and improving its skill over time by"
  },
  {
    "startMs": 188400,
    "text": "looking for"
  },
  {
    "startMs": 189800,
    "text": "um"
  },
  {
    "startMs": 190800,
    "text": "for mistakes it made or looking at"
  },
  {
    "startMs": 192160,
    "text": "feedback that the human has given it."
  },
  {
    "startMs": 194080,
    "text": "Um so, I'm going to show an example of"
  },
  {
    "startMs": 195720,
    "text": "that."
  },
  {
    "startMs": 196800,
    "text": "Um let me swap here. Cool. Um so, this"
  },
  {
    "startMs": 200800,
    "text": "is actually a skill that we've used"
  },
  {
    "startMs": 202160,
    "text": "internally at Warp. We've open-sourced"
  },
  {
    "startMs": 204239,
    "text": "our client repository and it's fully run"
  },
  {
    "startMs": 206600,
    "text": "as as a factory. And so, this is an"
  },
  {
    "startMs": 208959,
    "text": "example of an of a"
  },
  {
    "startMs": 211120,
    "text": "of a skill that's that's backing a"
  },
  {
    "startMs": 212760,
    "text": "triage agent that we have today."
  },
  {
    "startMs": 214800,
    "text": "The idea is that um"
  },
  {
    "startMs": 217320,
    "text": "you you get an issue that comes in on"
  },
  {
    "startMs": 219160,
    "text": "GitHub, and you want to automatically"
  },
  {
    "startMs": 221120,
    "text": "run an agent to figure out like what"
  },
  {
    "startMs": 223519,
    "text": "sorts of um"
  },
  {
    "startMs": 224880,
    "text": "like it what sort of information did the"
  },
  {
    "startMs": 226519,
    "text": "user not provide that is going to help"
  },
  {
    "startMs": 228200,
    "text": "us figure out if this is something that"
  },
  {
    "startMs": 229400,
    "text": "we should build, an existing issue that"
  },
  {
    "startMs": 231200,
    "text": "we might actually just want to dedupe"
  },
  {
    "startMs": 232440,
    "text": "against or something else."
  },
  {
    "startMs": 234440,
    "text": "Um"
  },
  {
    "startMs": 235840,
    "text": "and then we have a Sorry, let me just"
  },
  {
    "startMs": 237800,
    "text": "switch over here."
  },
  {
    "startMs": 239440,
    "text": "And then we have a workflow that"
  },
  {
    "startMs": 241600,
    "text": "actually like runs that that triage um"
  },
  {
    "startMs": 245040,
    "text": "agent."
  },
  {
    "startMs": 246160,
    "text": "Um this is a pretty simple workflow. It"
  },
  {
    "startMs": 247800,
    "text": "just Every time an an an issue gets"
  },
  {
    "startMs": 250040,
    "text": "submitted on the repo, this"
  },
  {
    "startMs": 251840,
    "text": "um this workflow runs. And then here's"
  },
  {
    "startMs": 254040,
    "text": "the actual outer loop agent, which is"
  },
  {
    "startMs": 255520,
    "text": "about looking at the the work that the"
  },
  {
    "startMs": 258079,
    "text": "triage agent has done and improving it"
  },
  {
    "startMs": 259880,
    "text": "over time. So, the idea is that the"
  },
  {
    "startMs": 262000,
    "text": "agent looks at the decisions that the"
  },
  {
    "startMs": 264320,
    "text": "triage the inner loop triage agent made,"
  },
  {
    "startMs": 266800,
    "text": "and it um collects feedback signals,"
  },
  {
    "startMs": 268840,
    "text": "which might be uh you know, thumbs up or"
  },
  {
    "startMs": 270960,
    "text": "thumbs down on the issues. Um it might"
  },
  {
    "startMs": 273680,
    "text": "be users commenting on the issue. It"
  },
  {
    "startMs": 275520,
    "text": "might be someone from Warp um an"
  },
  {
    "startMs": 277240,
    "text": "employee of Warp commenting on the"
  },
  {
    "startMs": 278520,
    "text": "issue, uh giving the agent feedback on"
  },
  {
    "startMs": 280640,
    "text": "what it did."
  },
  {
    "startMs": 282040,
    "text": "Um and then ultimately that leads to"
  },
  {
    "startMs": 284880,
    "text": "like some synthesis of of the of that um"
  },
  {
    "startMs": 288080,
    "text": "iteration. And so, we we end up"
  },
  {
    "startMs": 290520,
    "text": "basically updating our inner loop skill"
  },
  {
    "startMs": 293200,
    "text": "with this outer loop agents. Um and"
  },
  {
    "startMs": 295360,
    "text": "that's what leads to this"
  },
  {
    "startMs": 296160,
    "text": "self-improvement loop with skills."
  },
  {
    "startMs": 298000,
    "text": "What's really nice about this is that um"
  },
  {
    "startMs": 300240,
    "text": "the"
  },
  {
    "startMs": 301280,
    "text": "So, in step four, you'll see we we added"
  },
  {
    "startMs": 303120,
    "text": "the triage skill, and then we open a"
  },
  {
    "startMs": 304360,
    "text": "pull request. So, that means that um all"
  },
  {
    "startMs": 306760,
    "text": "of the improvements to the inner loop"
  },
  {
    "startMs": 308080,
    "text": "skill are going to be tracked through"
  },
  {
    "startMs": 310160,
    "text": "get. So, you get like full observability"
  },
  {
    "startMs": 312720,
    "text": "into like how that skill is transformed"
  },
  {
    "startMs": 314600,
    "text": "over time. And that it also means that"
  },
  {
    "startMs": 316760,
    "text": "um"
  },
  {
    "startMs": 317320,
    "text": "a human is actually going to review"
  },
  {
    "startMs": 318400,
    "text": "those updates to the skill. So, that"
  },
  {
    "startMs": 320000,
    "text": "way, you know, this this outer loop"
  },
  {
    "startMs": 321800,
    "text": "agent doesn't make a mistake and"
  },
  {
    "startMs": 323080,
    "text": "ultimately cause your triage agent to to"
  },
  {
    "startMs": 324760,
    "text": "actually perform worse."
  },
  {
    "startMs": 326320,
    "text": "Um"
  },
  {
    "startMs": 327919,
    "text": "yeah, so that's that's kind of what what"
  },
  {
    "startMs": 329440,
    "text": "I want to talk about for skills. Um"
  },
  {
    "startMs": 332400,
    "text": "let me talk about persistent memory now."
  },
  {
    "startMs": 335200,
    "text": "So, I'm going to go back to my deck."
  },
  {
    "startMs": 337360,
    "text": "Um"
  },
  {
    "startMs": 339280,
    "text": "cool. So, skills are great for"
  },
  {
    "startMs": 341000,
    "text": "remembering procedures, but what about"
  },
  {
    "startMs": 342840,
    "text": "everything else? So, like what what"
  },
  {
    "startMs": 344800,
    "text": "happens when your Sentry agent"
  },
  {
    "startMs": 346919,
    "text": "um looks for"
  },
  {
    "startMs": 348720,
    "text": "an issue and"
  },
  {
    "startMs": 350880,
    "text": "and finds an issue, does some work to"
  },
  {
    "startMs": 352480,
    "text": "like gather context about it, and fixes"
  },
  {
    "startMs": 355320,
    "text": "it, but then now you have you encounter"
  },
  {
    "startMs": 357800,
    "text": "a similar issue in the future."
  },
  {
    "startMs": 360400,
    "text": "Your century your agent might get lucky"
  },
  {
    "startMs": 362600,
    "text": "and like be able to figure out the same"
  },
  {
    "startMs": 365360,
    "text": "root cause that it did the first time"
  },
  {
    "startMs": 366640,
    "text": "around, but that's not guaranteed. And"
  },
  {
    "startMs": 368440,
    "text": "and even if it does, you're you've"
  },
  {
    "startMs": 370000,
    "text": "probably wasted a lot of tokens"
  },
  {
    "startMs": 371560,
    "text": "re-gathering context on an issue that"
  },
  {
    "startMs": 373120,
    "text": "you've already fixed before. So, that's"
  },
  {
    "startMs": 375280,
    "text": "where"
  },
  {
    "startMs": 376000,
    "text": "something like persistent memory comes"
  },
  {
    "startMs": 377440,
    "text": "in."
  },
  {
    "startMs": 378720,
    "text": "You can think of this as a fact store"
  },
  {
    "startMs": 380600,
    "text": "scoped to an agent. Your agent runs and"
  },
  {
    "startMs": 383280,
    "text": "similar to the skill improvement loop,"
  },
  {
    "startMs": 384600,
    "text": "you have an outer loop agent that's"
  },
  {
    "startMs": 386040,
    "text": "going to extract facts, learnings,"
  },
  {
    "startMs": 387520,
    "text": "outcomes"
  },
  {
    "startMs": 388919,
    "text": "from your from your inner loop agent."
  },
  {
    "startMs": 390720,
    "text": "So, that way future runs of the agent"
  },
  {
    "startMs": 392600,
    "text": "can lean on what's already been done in"
  },
  {
    "startMs": 393960,
    "text": "the past."
  },
  {
    "startMs": 395160,
    "text": "So, I'm going to show an example of"
  },
  {
    "startMs": 396600,
    "text": "this. Come back to the deck here."
  },
  {
    "startMs": 399320,
    "text": "So, in warp or in Oz rather, like I"
  },
  {
    "startMs": 402280,
    "text": "mentioned, we built out a a cloud agent"
  },
  {
    "startMs": 403680,
    "text": "platform where you can run agents in the"
  },
  {
    "startMs": 405960,
    "text": "cloud. And"
  },
  {
    "startMs": 408000,
    "text": "for all for all these agents, you can"
  },
  {
    "startMs": 409360,
    "text": "attach memory stores to them. Again, a"
  },
  {
    "startMs": 411280,
    "text": "memory a memory store is a collection of"
  },
  {
    "startMs": 413200,
    "text": "facts. In this case, I have a century"
  },
  {
    "startMs": 415880,
    "text": "agent that"
  },
  {
    "startMs": 417640,
    "text": "has collected some memories over time"
  },
  {
    "startMs": 420240,
    "text": "about root cause analysis that it's done"
  },
  {
    "startMs": 422600,
    "text": "in the past. What's cool is like you can"
  },
  {
    "startMs": 424360,
    "text": "actually version this memory, you can"
  },
  {
    "startMs": 425840,
    "text": "update it as a human, create new"
  },
  {
    "startMs": 427520,
    "text": "memories or delete them yourselves,"
  },
  {
    "startMs": 429600,
    "text": "but primarily this this memory creation"
  },
  {
    "startMs": 431640,
    "text": "loop is driven by agents. You can also"
  },
  {
    "startMs": 434440,
    "text": "see where these memories were sourced"
  },
  {
    "startMs": 436160,
    "text": "from. So,"
  },
  {
    "startMs": 437320,
    "text": "I can actually like open up the run here"
  },
  {
    "startMs": 439680,
    "text": "to see"
  },
  {
    "startMs": 440840,
    "text": "Oh, well."
  },
  {
    "startMs": 442400,
    "text": "I think this my IP address here keeps"
  },
  {
    "startMs": 444240,
    "text": "changing and this my our staging"
  },
  {
    "startMs": 445960,
    "text": "platform is gated to an IP address, but"
  },
  {
    "startMs": 448280,
    "text": "anyways,"
  },
  {
    "startMs": 449640,
    "text": "you should like"
  },
  {
    "startMs": 450840,
    "text": "you can see where those memories were"
  },
  {
    "startMs": 452480,
    "text": "sourced from. So, that way you can be"
  },
  {
    "startMs": 454000,
    "text": "like, oh, like maybe that wasn't"
  },
  {
    "startMs": 455640,
    "text": "actually an important memory or that was"
  },
  {
    "startMs": 456880,
    "text": "kind of like a local maxima that I don't"
  },
  {
    "startMs": 458240,
    "text": "want to include in my memory store. Um"
  },
  {
    "startMs": 461480,
    "text": "and then when your agents continue to"
  },
  {
    "startMs": 463680,
    "text": "run, so in this case I have the sentry"
  },
  {
    "startMs": 465480,
    "text": "agent that's"
  },
  {
    "startMs": 466560,
    "text": "um that's done a bunch of runs after"
  },
  {
    "startMs": 468080,
    "text": "it's created those memories, it actually"
  },
  {
    "startMs": 469760,
    "text": "can lean on those memories in the future"
  },
  {
    "startMs": 471960,
    "text": "um when it's making um"
  },
  {
    "startMs": 473720,
    "text": "when it's triaging new issues. So in"
  },
  {
    "startMs": 475240,
    "text": "this case, you'll notice that um"
  },
  {
    "startMs": 477400,
    "text": "this agent ran, it did a bunch of work"
  },
  {
    "startMs": 479240,
    "text": "to do some root cause analysis, and then"
  },
  {
    "startMs": 481120,
    "text": "it ultimately reported back on which"
  },
  {
    "startMs": 482760,
    "text": "memories it used"
  },
  {
    "startMs": 484160,
    "text": "um"
  },
  {
    "startMs": 485360,
    "text": "uh in this case it it it it was able to"
  },
  {
    "startMs": 487720,
    "text": "use those um five memories that I had in"
  },
  {
    "startMs": 489560,
    "text": "that memory sort to to speed up the work"
  },
  {
    "startMs": 491640,
    "text": "that I did this time around."
  },
  {
    "startMs": 493800,
    "text": "Um,"
  },
  {
    "startMs": 495320,
    "text": "cool. Uh"
  },
  {
    "startMs": 496640,
    "text": "let me go back to the deck."
  },
  {
    "startMs": 500400,
    "text": "So,"
  },
  {
    "startMs": 501440,
    "text": "what's cool in Oz is that persistent"
  },
  {
    "startMs": 503240,
    "text": "memory works across all harnesses, so"
  },
  {
    "startMs": 505360,
    "text": "Warp Zone, proprietary harness, plot"
  },
  {
    "startMs": 507160,
    "text": "code, Kodak, it doesn't really matter"
  },
  {
    "startMs": 508440,
    "text": "what you're running on Oz, um memories"
  },
  {
    "startMs": 510160,
    "text": "will automatically be created for you."
  },
  {
    "startMs": 512200,
    "text": "Um, but you also have the control to to"
  },
  {
    "startMs": 515200,
    "text": "review those memories, version them,"
  },
  {
    "startMs": 517360,
    "text": "delete them, or edit them as you please."
  },
  {
    "startMs": 519719,
    "text": "Um, all that being fully traceable."
  },
  {
    "startMs": 522919,
    "text": "Um, cool. That's what I wanted to say"
  },
  {
    "startMs": 524520,
    "text": "about memory. Um,"
  },
  {
    "startMs": 526640,
    "text": "last thing I want to talk about here is"
  },
  {
    "startMs": 528240,
    "text": "model routing. So, um you may be"
  },
  {
    "startMs": 530560,
    "text": "wondering how this plays into"
  },
  {
    "startMs": 531560,
    "text": "self-improvement loops for um factories."
  },
  {
    "startMs": 534040,
    "text": "Well, first I want to touch on why model"
  },
  {
    "startMs": 535360,
    "text": "routing is important in a factory in the"
  },
  {
    "startMs": 536600,
    "text": "first place."
  },
  {
    "startMs": 537640,
    "text": "Um, it can become prohibitively"
  },
  {
    "startMs": 539000,
    "text": "expensive to run all of your agents that"
  },
  {
    "startMs": 541200,
    "text": "are doing simple things like triage or"
  },
  {
    "startMs": 543280,
    "text": "fixing simple CI failures with Opus."
  },
  {
    "startMs": 545880,
    "text": "That's just like going to be really"
  },
  {
    "startMs": 547240,
    "text": "expensive over time, and I'm sure some"
  },
  {
    "startMs": 548920,
    "text": "of you have maybe been burnt by this or"
  },
  {
    "startMs": 550320,
    "text": "have heard stories of of or"
  },
  {
    "startMs": 552080,
    "text": "organizations being burnt by this. Um,"
  },
  {
    "startMs": 554320,
    "text": "so model like choosing which models you"
  },
  {
    "startMs": 556120,
    "text": "use for which tasks that is actually"
  },
  {
    "startMs": 558080,
    "text": "very important um and something that"
  },
  {
    "startMs": 559680,
    "text": "we're we've been working on at work."
  },
  {
    "startMs": 561520,
    "text": "Um, so in Warp, you can actually use uh"
  },
  {
    "startMs": 564080,
    "text": "our out-of-the-box model routers, we"
  },
  {
    "startMs": 565560,
    "text": "call them our auto models. Um, over like"
  },
  {
    "startMs": 568320,
    "text": "as new models come out, we're all we're"
  },
  {
    "startMs": 569680,
    "text": "always evaluating like what are the best"
  },
  {
    "startMs": 571640,
    "text": "models when it comes to Pareto"
  },
  {
    "startMs": 572920,
    "text": "efficiency. And so, like if you're a new"
  },
  {
    "startMs": 575120,
    "text": "user and you just want to get started,"
  },
  {
    "startMs": 576920,
    "text": "it's much better than like pinning your"
  },
  {
    "startMs": 578320,
    "text": "model to like Opus or or or Haiku, for"
  },
  {
    "startMs": 581120,
    "text": "example. Um"
  },
  {
    "startMs": 584120,
    "text": "let me show you all um that experience"
  },
  {
    "startMs": 586960,
    "text": "in Warp."
  },
  {
    "startMs": 587920,
    "text": "So, let me go here."
  },
  {
    "startMs": 593080,
    "text": "Give me 1 sec."
  },
  {
    "startMs": 594640,
    "text": "Got to drag this over here."
  },
  {
    "startMs": 596920,
    "text": "So, here I have my Warp desktop um and"
  },
  {
    "startMs": 599440,
    "text": "you can see that um alongside being able"
  },
  {
    "startMs": 601720,
    "text": "to define models um"
  },
  {
    "startMs": 604840,
    "text": "being able to use Warp's auto models is"
  },
  {
    "startMs": 606400,
    "text": "one option, like these models here, but"
  },
  {
    "startMs": 608760,
    "text": "you can also define your own model"
  },
  {
    "startMs": 610160,
    "text": "routed um"
  },
  {
    "startMs": 612160,
    "text": "uh models, which uh basically boil down"
  },
  {
    "startMs": 615040,
    "text": "to like a set of rules that you can give"
  },
  {
    "startMs": 616960,
    "text": "um to us and we will like make decisions"
  },
  {
    "startMs": 619440,
    "text": "about model routing based on those"
  },
  {
    "startMs": 620960,
    "text": "rules. So, here I have an example of a"
  },
  {
    "startMs": 623000,
    "text": "configuration file that I've defined um"
  },
  {
    "startMs": 625560,
    "text": "where I've defined a few different rules"
  },
  {
    "startMs": 626960,
    "text": "about like I want database migrations"
  },
  {
    "startMs": 629000,
    "text": "done with with GLM fire with GLM or I"
  },
  {
    "startMs": 631720,
    "text": "want um you know, uh runbooks and API"
  },
  {
    "startMs": 634760,
    "text": "documentation done with Quen. Um"
  },
  {
    "startMs": 638160,
    "text": "So, this really allows you to like like"
  },
  {
    "startMs": 639760,
    "text": "to define classes of tasks and be able"
  },
  {
    "startMs": 642600,
    "text": "to say, \"Okay, I want this this class of"
  },
  {
    "startMs": 644560,
    "text": "task done by this model because that's"
  },
  {
    "startMs": 646440,
    "text": "kind of what you've seen.\""
  },
  {
    "startMs": 647960,
    "text": "At this point though, that's kind of"
  },
  {
    "startMs": 649320,
    "text": "more of an art than a science um and so"
  },
  {
    "startMs": 652280,
    "text": "uh what we're working on next is being"
  },
  {
    "startMs": 653880,
    "text": "able to define evals for um"
  },
  {
    "startMs": 656920,
    "text": "customer-facing evals where you can"
  },
  {
    "startMs": 658600,
    "text": "actually define what sorts of things you"
  },
  {
    "startMs": 660720,
    "text": "care about and what knobs you want to"
  },
  {
    "startMs": 662040,
    "text": "turn on these configurations and"
  },
  {
    "startMs": 664280,
    "text": "actually see, \"Okay, is is Haiku better"
  },
  {
    "startMs": 667160,
    "text": "than fi- than GLM than Opus in a"
  },
  {
    "startMs": 670000,
    "text": "specific class of issues?\" That's like"
  },
  {
    "startMs": 671760,
    "text": "specific to your workflows and not just"
  },
  {
    "startMs": 673720,
    "text": "some like generic benchmark that um that"
  },
  {
    "startMs": 676520,
    "text": "someone posted online."
  },
  {
    "startMs": 679120,
    "text": "Um let me go back here. Give me a sec."
  },
  {
    "startMs": 684760,
    "text": "Cool."
  },
  {
    "startMs": 690720,
    "text": "Cool. Um"
  },
  {
    "startMs": 693800,
    "text": "and yeah, so let me uh I'm in my"
  },
  {
    "startMs": 695960,
    "text": "slideshow."
  },
  {
    "startMs": 702880,
    "text": "So, here is kind of like a a high-level"
  },
  {
    "startMs": 704920,
    "text": "overview of how this works in Warp"
  },
  {
    "startMs": 706120,
    "text": "today. So, um internally we have our"
  },
  {
    "startMs": 708800,
    "text": "model routing rules, um but then we have"
  },
  {
    "startMs": 711000,
    "text": "like an eval sidecar that's determining"
  },
  {
    "startMs": 713520,
    "text": "um what sorts of tasks are actually"
  },
  {
    "startMs": 715800,
    "text": "better with which models. And we do this"
  },
  {
    "startMs": 716960,
    "text": "with a simple like best-at-k um approach"
  },
  {
    "startMs": 719760,
    "text": "where when you type a prompt, we run a"
  },
  {
    "startMs": 721400,
    "text": "bunch of agents in Oz uh across a"
  },
  {
    "startMs": 723240,
    "text": "different set of models. Um and we found"
  },
  {
    "startMs": 725600,
    "text": "success in like being able to determine,"
  },
  {
    "startMs": 726960,
    "text": "\"Oh, like UI tasks are really well done"
  },
  {
    "startMs": 729280,
    "text": "with GLM. Um we don't really need to run"
  },
  {
    "startMs": 731120,
    "text": "those with Opus. It's much more"
  },
  {
    "startMs": 732080,
    "text": "efficient to run those with GLM.\""
  },
  {
    "startMs": 734000,
    "text": "Um so, that's how we've been using it,"
  },
  {
    "startMs": 735320,
    "text": "but um sooner enough we're going to be"
  },
  {
    "startMs": 736840,
    "text": "building this into our product so that"
  },
  {
    "startMs": 738000,
    "text": "way our our customers can actually use"
  },
  {
    "startMs": 740000,
    "text": "it in their own workflows. Um"
  },
  {
    "startMs": 743480,
    "text": "Yeah, that's uh that's mostly what I had"
  },
  {
    "startMs": 745160,
    "text": "today. Um"
  },
  {
    "startMs": 746480,
    "text": "we're around in a booth around that"
  },
  {
    "startMs": 748400,
    "text": "side. So, if you're interested in this"
  },
  {
    "startMs": 749800,
    "text": "kind of stuff, come talk to me. I'm"
  },
  {
    "startMs": 751560,
    "text": "always down to talk about this kind of"
  },
  {
    "startMs": 752680,
    "text": "stuff. But yeah, thanks for your time"
  },
  {
    "startMs": 754240,
    "text": "today. That's all I had."
  },
  {
    "startMs": 769633,
    "text": ">> [music]"
  }
]