[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12960,
    "text": "All right. Um, so again, I'm my name is"
  },
  {
    "startMs": 16400,
    "text": "Potato. Uh, I work on Graphbot at"
  },
  {
    "startMs": 19680,
    "text": "SpaceXsi."
  },
  {
    "startMs": 21199,
    "text": "And can I ask you a quick show of hands?"
  },
  {
    "startMs": 23920,
    "text": "How many of you have ever looked at code"
  },
  {
    "startMs": 26000,
    "text": "that somebody else wrote and you thought"
  },
  {
    "startMs": 28560,
    "text": "that's all slop?"
  },
  {
    "startMs": 31760,
    "text": "Well, I have too."
  },
  {
    "startMs": 35280,
    "text": "But today, I'm going to convince you"
  },
  {
    "startMs": 36640,
    "text": "that you can turn that slop into gold."
  },
  {
    "startMs": 42559,
    "text": "Uh, and so I did something crazy last"
  },
  {
    "startMs": 44559,
    "text": "month and I shipped 3,000 ploy requests"
  },
  {
    "startMs": 47039,
    "text": "to production. Now, before agents, we"
  },
  {
    "startMs": 49760,
    "text": "never really cared about volume. uh"
  },
  {
    "startMs": 52000,
    "text": "except for the outliers."
  },
  {
    "startMs": 54559,
    "text": "And at my previous company, we call"
  },
  {
    "startMs": 56559,
    "text": "these people coding machines. People"
  },
  {
    "startMs": 58559,
    "text": "like Brett Taylor, who famously rewrote"
  },
  {
    "startMs": 60800,
    "text": "all of Google Maps uh in a weekend, just"
  },
  {
    "startMs": 63680,
    "text": "an insanely productive engineer. Uh and"
  },
  {
    "startMs": 66720,
    "text": "well, with agents, anyone can now be a"
  },
  {
    "startMs": 68880,
    "text": "coding machine. And my thesis is that"
  },
  {
    "startMs": 71439,
    "text": "the job of a software engineer, the meta"
  },
  {
    "startMs": 73760,
    "text": "of being a software engineer is figuring"
  },
  {
    "startMs": 75760,
    "text": "out how to maintain or even improve the"
  },
  {
    "startMs": 78560,
    "text": "quality of your code while massively"
  },
  {
    "startMs": 80720,
    "text": "increasing your productivity."
  },
  {
    "startMs": 83680,
    "text": "To get there, some people like to call"
  },
  {
    "startMs": 85200,
    "text": "this a software factory. Uh, but I like"
  },
  {
    "startMs": 87759,
    "text": "to keep it simple and I think of it as"
  },
  {
    "startMs": 90240,
    "text": "all about trust."
  },
  {
    "startMs": 92960,
    "text": "not blind trust, but investing in your"
  },
  {
    "startMs": 95360,
    "text": "agents environment so you can trust that"
  },
  {
    "startMs": 97520,
    "text": "your agents are going to do the right"
  },
  {
    "startMs": 99119,
    "text": "thing. Now, it takes quite a lot of"
  },
  {
    "startMs": 102640,
    "text": "effort to get there. Um, and it took me"
  },
  {
    "startMs": 105280,
    "text": "6 months to get to a point where I could"
  },
  {
    "startMs": 107680,
    "text": "ship thousands of pull requests, a lot"
  },
  {
    "startMs": 109840,
    "text": "of them in my sleep, but it's certainly"
  },
  {
    "startMs": 112240,
    "text": "possible and it will take you much less"
  },
  {
    "startMs": 114799,
    "text": "time than I did. So, let's start by"
  },
  {
    "startMs": 117200,
    "text": "talking about how we move from a low"
  },
  {
    "startMs": 119119,
    "text": "trust environment to a high trust one."
  },
  {
    "startMs": 123280,
    "text": "So this inverse pyramid is how I think"
  },
  {
    "startMs": 125360,
    "text": "about where to spend time on your agents"
  },
  {
    "startMs": 128000,
    "text": "environment."
  },
  {
    "startMs": 130000,
    "text": "The first two layers are deterministic."
  },
  {
    "startMs": 132480,
    "text": "So using your code itself. So things"
  },
  {
    "startMs": 134800,
    "text": "like good API design architecture"
  },
  {
    "startMs": 137440,
    "text": "refactoring or using lit rules, type"
  },
  {
    "startMs": 140400,
    "text": "systems, compilers or you know different"
  },
  {
    "startMs": 142879,
    "text": "kinds of testing. And you can use both"
  },
  {
    "startMs": 145440,
    "text": "of these layers together to constrain"
  },
  {
    "startMs": 147520,
    "text": "what agents can do and force them to do"
  },
  {
    "startMs": 150000,
    "text": "the right thing by default."
  },
  {
    "startMs": 152879,
    "text": "Verification, both runtime and formal,"
  },
  {
    "startMs": 155200,
    "text": "are great tools to layer on to find bugs"
  },
  {
    "startMs": 158160,
    "text": "and check properties and the invariance"
  },
  {
    "startMs": 160160,
    "text": "in your system that you want to hold"
  },
  {
    "startMs": 161760,
    "text": "true and also help you check whether or"
  },
  {
    "startMs": 164800,
    "text": "not the agent understood your goals and"
  },
  {
    "startMs": 166879,
    "text": "your intent."
  },
  {
    "startMs": 168959,
    "text": "And then finally, you want to layer with"
  },
  {
    "startMs": 170800,
    "text": "a little bit of pros. So things like"
  },
  {
    "startMs": 172800,
    "text": "rules, skills, agents.mmd, things that"
  },
  {
    "startMs": 176560,
    "text": "require judgment."
  },
  {
    "startMs": 179360,
    "text": "But let's start with your codebase."
  },
  {
    "startMs": 181200,
    "text": "Agents generate their outputs based on"
  },
  {
    "startMs": 183519,
    "text": "their inputs. And so the the quality of"
  },
  {
    "startMs": 186239,
    "text": "the input really matters. And it just so"
  },
  {
    "startMs": 188159,
    "text": "happens that your codebase is one of the"
  },
  {
    "startMs": 190640,
    "text": "largest inputs. So whenever you're"
  },
  {
    "startMs": 193360,
    "text": "correcting your agent, think about how"
  },
  {
    "startMs": 195200,
    "text": "you can make that a deterministic check"
  },
  {
    "startMs": 197440,
    "text": "instead of adding another line to your"
  },
  {
    "startMs": 199840,
    "text": "rules or your agents.mmd because it's"
  },
  {
    "startMs": 202239,
    "text": "far more reproducible when it's in code"
  },
  {
    "startMs": 204319,
    "text": "and it lets you trust it more."
  },
  {
    "startMs": 207440,
    "text": "And the analogy that I like to use is"
  },
  {
    "startMs": 209440,
    "text": "that your codebase is kind of like a"
  },
  {
    "startMs": 211599,
    "text": "garden. You know, code used to arrive a"
  },
  {
    "startMs": 214239,
    "text": "few pull requests at a time. And uh I"
  },
  {
    "startMs": 217360,
    "text": "remember going into work in 2024 and"
  },
  {
    "startMs": 220400,
    "text": "reviewing the three pull requests that"
  },
  {
    "startMs": 222159,
    "text": "my colleagues wrote colleagues wrote and"
  },
  {
    "startMs": 224480,
    "text": "today that number is more like 300. So"
  },
  {
    "startMs": 228400,
    "text": "uh you know every engineer is now a team"
  },
  {
    "startMs": 230480,
    "text": "of hundreds or even thousands of agents"
  },
  {
    "startMs": 233040,
    "text": "and code is arriving so rapidly that it"
  },
  {
    "startMs": 236319,
    "text": "almost feels like a continuous stream"
  },
  {
    "startMs": 238959,
    "text": "and it almost feels organic like a"
  },
  {
    "startMs": 241599,
    "text": "living thing."
  },
  {
    "startMs": 243439,
    "text": "And in this analogy, your codebase, the"
  },
  {
    "startMs": 246400,
    "text": "constraints that you impose, the"
  },
  {
    "startMs": 248319,
    "text": "guardrails that you put up are the"
  },
  {
    "startMs": 250400,
    "text": "concrete foundations that the new code"
  },
  {
    "startMs": 252799,
    "text": "that your agents write grow around just"
  },
  {
    "startMs": 255599,
    "text": "like uh the building in this picture."
  },
  {
    "startMs": 258079,
    "text": "And so in many ways, the job of being a"
  },
  {
    "startMs": 259759,
    "text": "software engineer is kind of becoming"
  },
  {
    "startMs": 261840,
    "text": "like gardening"
  },
  {
    "startMs": 264880,
    "text": "because code is the biggest input for"
  },
  {
    "startMs": 266639,
    "text": "agents. It means that any workarounds,"
  },
  {
    "startMs": 269680,
    "text": "um, anti-atterns, deprecated code, tech,"
  },
  {
    "startMs": 273520,
    "text": "it's now likely to be just copied by the"
  },
  {
    "startMs": 275759,
    "text": "next agent since it looks like the"
  },
  {
    "startMs": 277600,
    "text": "established pattern in your codebase."
  },
  {
    "startMs": 280000,
    "text": "And uh, it's like you have some weeds or"
  },
  {
    "startMs": 283440,
    "text": "pests growing in your garden and then"
  },
  {
    "startMs": 285520,
    "text": "they spread everywhere before you know"
  },
  {
    "startMs": 287199,
    "text": "it."
  },
  {
    "startMs": 291120,
    "text": "So you want to make your codebase as"
  },
  {
    "startMs": 293120,
    "text": "pristine as possible so that agents are"
  },
  {
    "startMs": 295600,
    "text": "always copying the good stuff. All"
  },
  {
    "startMs": 298240,
    "text": "right, let's look at some real examples."
  },
  {
    "startMs": 300639,
    "text": "Uh so Dune is a client framework that we"
  },
  {
    "startMs": 303040,
    "text": "created for Grabbot to implement this"
  },
  {
    "startMs": 305199,
    "text": "idea for organic architecture."
  },
  {
    "startMs": 308080,
    "text": "It's not really that interesting"
  },
  {
    "startMs": 309360,
    "text": "technically. Uh you can kind of think of"
  },
  {
    "startMs": 311440,
    "text": "Dune as um kind of like Nex.js for"
  },
  {
    "startMs": 314400,
    "text": "Electron apps. But the interesting part"
  },
  {
    "startMs": 316639,
    "text": "of Dune is that it has a lot of"
  },
  {
    "startMs": 318160,
    "text": "constraints and lit rules that we've"
  },
  {
    "startMs": 320400,
    "text": "encoded uh a lot of the lessons that"
  },
  {
    "startMs": 322639,
    "text": "we've learned building desktop"
  },
  {
    "startMs": 324479,
    "text": "applications."
  },
  {
    "startMs": 326560,
    "text": "And one such lesson is that anything"
  },
  {
    "startMs": 328880,
    "text": "that requires nuance to use well often"
  },
  {
    "startMs": 331520,
    "text": "leads to slop. Uh you know your classic"
  },
  {
    "startMs": 334400,
    "text": "example is uh you know use effect in"
  },
  {
    "startMs": 336800,
    "text": "React, right? It has its use cases uh"
  },
  {
    "startMs": 339840,
    "text": "but it's often misused. You know some"
  },
  {
    "startMs": 341919,
    "text": "people joke that you need a PhD to use"
  },
  {
    "startMs": 343840,
    "text": "use effect. Um and in in other words, it"
  },
  {
    "startMs": 346960,
    "text": "really requires a lot of nuance to use B"
  },
  {
    "startMs": 350080,
    "text": "and agents are not good at using it. Um"
  },
  {
    "startMs": 353360,
    "text": "so our solution has been to just ban it"
  },
  {
    "startMs": 356000,
    "text": "in product code and it's really only"
  },
  {
    "startMs": 358240,
    "text": "allowed in some places inside of Dune"
  },
  {
    "startMs": 360479,
    "text": "APIs"
  },
  {
    "startMs": 362320,
    "text": "and effectively this problem has now"
  },
  {
    "startMs": 364639,
    "text": "been categorically eliminated and we"
  },
  {
    "startMs": 367039,
    "text": "never need to catch that during code"
  },
  {
    "startMs": 368560,
    "text": "review again."
  },
  {
    "startMs": 370800,
    "text": "Another interesting pattern that we use"
  },
  {
    "startMs": 372400,
    "text": "is context through directory structure."
  },
  {
    "startMs": 375120,
    "text": "Uh so one anti-attern that we saw with"
  },
  {
    "startMs": 377520,
    "text": "cursor which is an electron app is that"
  },
  {
    "startMs": 380000,
    "text": "agents would often import main thread"
  },
  {
    "startMs": 382880,
    "text": "code into the renderer thread which"
  },
  {
    "startMs": 384639,
    "text": "would cause long tasks and jank."
  },
  {
    "startMs": 388080,
    "text": "So in uh grapot and in dunes we've made"
  },
  {
    "startMs": 391039,
    "text": "this very very conventional. Uh we have"
  },
  {
    "startMs": 393360,
    "text": "these two top level directories"
  },
  {
    "startMs": 395280,
    "text": "literally called electron renderer and"
  },
  {
    "startMs": 397199,
    "text": "electron main and imports between them"
  },
  {
    "startMs": 400560,
    "text": "are forbidden or banned by our"
  },
  {
    "startMs": 402720,
    "text": "dependency graph analyzer and some lint"
  },
  {
    "startMs": 405039,
    "text": "rules. And again you know these are"
  },
  {
    "startMs": 407600,
    "text": "simple lint rules to set up but it means"
  },
  {
    "startMs": 409840,
    "text": "one less thing that we need to review"
  },
  {
    "startMs": 411759,
    "text": "and think about uh you know during code"
  },
  {
    "startMs": 414400,
    "text": "review."
  },
  {
    "startMs": 416880,
    "text": "And then one more surprising thing that"
  },
  {
    "startMs": 419120,
    "text": "I've seen is agents using markdown. So"
  },
  {
    "startMs": 422240,
    "text": "things like documentation, uh, code"
  },
  {
    "startMs": 424639,
    "text": "comments, rules or skills, and they use"
  },
  {
    "startMs": 427199,
    "text": "this as a place to write down their"
  },
  {
    "startMs": 429680,
    "text": "justifications for not doing the right"
  },
  {
    "startMs": 431599,
    "text": "thing, or I don't know if you've ever"
  },
  {
    "startMs": 433440,
    "text": "seen this, but they'll over rotate on"
  },
  {
    "startMs": 435440,
    "text": "some small correction that you made, and"
  },
  {
    "startMs": 437520,
    "text": "they treat it like a hard rule that can"
  },
  {
    "startMs": 439360,
    "text": "never be broken ever again."
  },
  {
    "startMs": 442720,
    "text": "And here's another example from cursor."
  },
  {
    "startMs": 445120,
    "text": "So agents really loved using imperative"
  },
  {
    "startMs": 447680,
    "text": "timers. So things like set timeout,"
  },
  {
    "startMs": 449759,
    "text": "request animation frame, request idle"
  },
  {
    "startMs": 451680,
    "text": "call back. And they were using these as"
  },
  {
    "startMs": 453680,
    "text": "quick fixes because they were really"
  },
  {
    "startMs": 456400,
    "text": "patching over an architecture clash"
  },
  {
    "startMs": 458400,
    "text": "between uh VS Code's imperative style"
  },
  {
    "startMs": 461599,
    "text": "and React's declarative style."
  },
  {
    "startMs": 464400,
    "text": "So our choice of architecture actually"
  },
  {
    "startMs": 466639,
    "text": "made both the client and the agent"
  },
  {
    "startMs": 469599,
    "text": "performance much worse. And if you've"
  },
  {
    "startMs": 472000,
    "text": "ever done any performance work in"
  },
  {
    "startMs": 473199,
    "text": "JavaScript, you'll know that when you"
  },
  {
    "startMs": 474479,
    "text": "have tons of timers being rapidly"
  },
  {
    "startMs": 476639,
    "text": "created and destroyed causes a lot of uh"
  },
  {
    "startMs": 479520,
    "text": "major garbage collection, which causes a"
  },
  {
    "startMs": 481840,
    "text": "lot of lag and jen. Uh but the worst"
  },
  {
    "startMs": 484240,
    "text": "part is that agents wrote a lot of code"
  },
  {
    "startMs": 486400,
    "text": "comments like you can see in red here"
  },
  {
    "startMs": 488720,
    "text": "explaining why this quick fix was"
  },
  {
    "startMs": 490720,
    "text": "justified and before you know it, we had"
  },
  {
    "startMs": 494000,
    "text": "timers spreading everywhere and the you"
  },
  {
    "startMs": 496560,
    "text": "know the performance of the app really"
  },
  {
    "startMs": 498000,
    "text": "suffered."
  },
  {
    "startMs": 500560,
    "text": "Now in graphbot uh we learned our lesson"
  },
  {
    "startMs": 503039,
    "text": "there and we decided to just ban 99% of"
  },
  {
    "startMs": 505840,
    "text": "comments uh because they go stale easily"
  },
  {
    "startMs": 509280,
    "text": "uh you know humans just really don't"
  },
  {
    "startMs": 510879,
    "text": "bother reading the comment slop anyway"
  },
  {
    "startMs": 513360,
    "text": "and again it gives an agent the an"
  },
  {
    "startMs": 516000,
    "text": "escape hatch to justify not doing the"
  },
  {
    "startMs": 518719,
    "text": "right thing because it's hard. Um and"
  },
  {
    "startMs": 521839,
    "text": "the worst part about comments though is"
  },
  {
    "startMs": 523440,
    "text": "that they're not enforcable. So what"
  },
  {
    "startMs": 525920,
    "text": "seems well-intentioned, you know, like"
  },
  {
    "startMs": 528000,
    "text": "I'm going to tell the the next agent not"
  },
  {
    "startMs": 530080,
    "text": "to do something, never ends up actually"
  },
  {
    "startMs": 532399,
    "text": "being anything that you can trust. So in"
  },
  {
    "startMs": 536080,
    "text": "Grogbot and Dune and in Dune, uh we now"
  },
  {
    "startMs": 538959,
    "text": "have lint rules that give uh good and"
  },
  {
    "startMs": 541680,
    "text": "very helpful error messages telling the"
  },
  {
    "startMs": 543680,
    "text": "agent to fix the root cause, the real"
  },
  {
    "startMs": 546399,
    "text": "problem, uh rather than just patch"
  },
  {
    "startMs": 548399,
    "text": "around it."
  },
  {
    "startMs": 550560,
    "text": "And so our general principle is that"
  },
  {
    "startMs": 552240,
    "text": "every time we correct our agents, it"
  },
  {
    "startMs": 553760,
    "text": "turns into one of these deterministic"
  },
  {
    "startMs": 555440,
    "text": "checks or we think about how to"
  },
  {
    "startMs": 557360,
    "text": "categorically eliminate that problem in"
  },
  {
    "startMs": 559200,
    "text": "the code through better API design,"
  },
  {
    "startMs": 561600,
    "text": "through better architecture. And if you"
  },
  {
    "startMs": 564640,
    "text": "do this enough times, you'll find that"
  },
  {
    "startMs": 567360,
    "text": "you have just less things to look out"
  },
  {
    "startMs": 569200,
    "text": "for in code review. And at some point"
  },
  {
    "startMs": 571440,
    "text": "you realize there's there's nothing left"
  },
  {
    "startMs": 573440,
    "text": "to review and you can just let agents"
  },
  {
    "startMs": 575680,
    "text": "merge their most of their own pull"
  },
  {
    "startMs": 578080,
    "text": "requests."
  },
  {
    "startMs": 580880,
    "text": "All right. Uh moving on. Verification is"
  },
  {
    "startMs": 583600,
    "text": "also a really big part of how we make"
  },
  {
    "startMs": 585519,
    "text": "our agents more autonomous and also"
  },
  {
    "startMs": 588000,
    "text": "increase our confidence and trust in"
  },
  {
    "startMs": 589839,
    "text": "their work. And our code review process"
  },
  {
    "startMs": 592320,
    "text": "has now really shrunk into looking at"
  },
  {
    "startMs": 594080,
    "text": "verification artifacts instead. So"
  },
  {
    "startMs": 596480,
    "text": "things like videos, screenshots,"
  },
  {
    "startMs": 598880,
    "text": "diagrams, formal verification."
  },
  {
    "startMs": 602000,
    "text": "Um, and the most basic example of this"
  },
  {
    "startMs": 604000,
    "text": "is runtime verification. Uh, which is"
  },
  {
    "startMs": 606480,
    "text": "just a fancy way of saying giving your"
  },
  {
    "startMs": 608320,
    "text": "agent a way to run the code for itself"
  },
  {
    "startMs": 610560,
    "text": "and use debugging tools that you know,"
  },
  {
    "startMs": 612640,
    "text": "human developers use as well, like the"
  },
  {
    "startMs": 614560,
    "text": "Chrome DevTools protocol. Um, in our"
  },
  {
    "startMs": 617360,
    "text": "runtime verification skills, we have"
  },
  {
    "startMs": 619200,
    "text": "this thing that we call a feature map."
  },
  {
    "startMs": 621680,
    "text": "And that's basically kind of like a site"
  },
  {
    "startMs": 624000,
    "text": "map of all of the features that are"
  },
  {
    "startMs": 625839,
    "text": "available in your application from the"
  },
  {
    "startMs": 627600,
    "text": "user's point of view. And so with this,"
  },
  {
    "startMs": 630079,
    "text": "agents don't need to guess when you say,"
  },
  {
    "startMs": 632160,
    "text": "you know, the thingy in the left"
  },
  {
    "startMs": 633440,
    "text": "sidebar. It gives you some shared"
  },
  {
    "startMs": 635360,
    "text": "vocabulary."
  },
  {
    "startMs": 637600,
    "text": "And we also make use of custom CLIs"
  },
  {
    "startMs": 639680,
    "text": "inside these skills so that uh agents"
  },
  {
    "startMs": 642160,
    "text": "can reuse the the CLI and the driver"
  },
  {
    "startMs": 645120,
    "text": "every time which cuts down on token"
  },
  {
    "startMs": 646720,
    "text": "usage and is also testable."
  },
  {
    "startMs": 650720,
    "text": "Um and then recently we've also started"
  },
  {
    "startMs": 652480,
    "text": "looking at formal verification. Uh so we"
  },
  {
    "startMs": 655040,
    "text": "now use TA plus to do some model"
  },
  {
    "startMs": 657200,
    "text": "checking of major user flows such as uh"
  },
  {
    "startMs": 659839,
    "text": "notifications across multiple devices"
  },
  {
    "startMs": 661680,
    "text": "and platforms. Uh we use a lean as well"
  },
  {
    "startMs": 664399,
    "text": "for proving certain algorithms. Uh for"
  },
  {
    "startMs": 666880,
    "text": "example, we have our own we wrote our"
  },
  {
    "startMs": 668720,
    "text": "own custom virtualization library and we"
  },
  {
    "startMs": 670720,
    "text": "use lean there to check that we don't"
  },
  {
    "startMs": 672640,
    "text": "have you know flickering or jittering"
  },
  {
    "startMs": 674800,
    "text": "problems. Um and we're still"
  },
  {
    "startMs": 676959,
    "text": "experimenting with this approach."
  },
  {
    "startMs": 678160,
    "text": "Overall it seems very promising. Uh"
  },
  {
    "startMs": 680480,
    "text": "we've already found quite a few bugs uh"
  },
  {
    "startMs": 682399,
    "text": "with formal verification."
  },
  {
    "startMs": 685200,
    "text": "Um but it's certainly not a silver"
  },
  {
    "startMs": 686880,
    "text": "bullet and we're treating it as another"
  },
  {
    "startMs": 690160,
    "text": "tool, another layer in our agents uh"
  },
  {
    "startMs": 693440,
    "text": "environment."
  },
  {
    "startMs": 695600,
    "text": "Uh and then this is a very fast video"
  },
  {
    "startMs": 697839,
    "text": "but uh another thing that we just"
  },
  {
    "startMs": 699600,
    "text": "started looking at is property based"
  },
  {
    "startMs": 701279,
    "text": "testing as well for UIs. Uh so we use"
  },
  {
    "startMs": 704000,
    "text": "this library called Bombadil. Um and"
  },
  {
    "startMs": 706320,
    "text": "this has also caught a few UI bugs for"
  },
  {
    "startMs": 708560,
    "text": "us already."
  },
  {
    "startMs": 711920,
    "text": "Okay. Okay, so zooming out, I saw this"
  },
  {
    "startMs": 713519,
    "text": "tweet from Cindy and six months ago she"
  },
  {
    "startMs": 716240,
    "text": "she said that she was or she was talking"
  },
  {
    "startMs": 718800,
    "text": "about not being able to trust her agents"
  },
  {
    "startMs": 721040,
    "text": "because of subtle bugs that were getting"
  },
  {
    "startMs": 723040,
    "text": "merged."
  },
  {
    "startMs": 725120,
    "text": "And really what was missing for her in"
  },
  {
    "startMs": 726959,
    "text": "being able to trust her agents was the"
  },
  {
    "startMs": 728560,
    "text": "verification process. And now 6 months"
  },
  {
    "startMs": 731200,
    "text": "later, she says that 70% of her daily"
  },
  {
    "startMs": 733760,
    "text": "tokens go into verification. And I'd say"
  },
  {
    "startMs": 736320,
    "text": "that that tracks with what I see as"
  },
  {
    "startMs": 738480,
    "text": "well, that most of my token usage goes"
  },
  {
    "startMs": 740560,
    "text": "to verification."
  },
  {
    "startMs": 742480,
    "text": "And if you really look closely at our"
  },
  {
    "startMs": 743920,
    "text": "process, none of these things here are"
  },
  {
    "startMs": 746240,
    "text": "new or novel techniques that we've never"
  },
  {
    "startMs": 748399,
    "text": "had before."
  },
  {
    "startMs": 750639,
    "text": "These are just good software engineering"
  },
  {
    "startMs": 752399,
    "text": "tools, but ones that might have been"
  },
  {
    "startMs": 754800,
    "text": "prohibitively expensive to use before as"
  },
  {
    "startMs": 757279,
    "text": "they were hard to understand or write or"
  },
  {
    "startMs": 759440,
    "text": "maintain. But agents drastically cut"
  },
  {
    "startMs": 762800,
    "text": "this cost down. And really, we're"
  },
  {
    "startMs": 764560,
    "text": "entering a golden age of software"
  },
  {
    "startMs": 766399,
    "text": "testing and verification because agents"
  },
  {
    "startMs": 768880,
    "text": "let us test everything to a much greater"
  },
  {
    "startMs": 770880,
    "text": "degree than used to be economical."
  },
  {
    "startMs": 774959,
    "text": "And then formal verification can also"
  },
  {
    "startMs": 776880,
    "text": "help you better reason about how your"
  },
  {
    "startMs": 778399,
    "text": "product should work and whether your"
  },
  {
    "startMs": 780000,
    "text": "agents really understand your intent and"
  },
  {
    "startMs": 781680,
    "text": "goals. Uh I made this new skill called"
  },
  {
    "startMs": 784240,
    "text": "state chart uh in my plug-in PSAC potato"
  },
  {
    "startMs": 787440,
    "text": "stack uh which helps you visualize state"
  },
  {
    "startMs": 789519,
    "text": "machines from uh from the user's point"
  },
  {
    "startMs": 792240,
    "text": "of view. Uh, and it's I find this much"
  },
  {
    "startMs": 794399,
    "text": "easier to review alongside the videos"
  },
  {
    "startMs": 796399,
    "text": "and screenshots that we provide. Um, and"
  },
  {
    "startMs": 799360,
    "text": "so that's been great. Sorry for the"
  },
  {
    "startMs": 801680,
    "text": "flashbang. Uh, okay. Now, let's look at"
  },
  {
    "startMs": 804720,
    "text": "what the promised land looks like. What"
  },
  {
    "startMs": 807040,
    "text": "do you get in return for all of this,"
  },
  {
    "startMs": 809680,
    "text": "uh, you know, investment?"
  },
  {
    "startMs": 812240,
    "text": "So, on the Graphbot team, our velocity"
  },
  {
    "startMs": 814639,
    "text": "is uh, taking off and rapidly"
  },
  {
    "startMs": 816639,
    "text": "increasing. And last month, our team"
  },
  {
    "startMs": 819040,
    "text": "shipped a ridiculous 11,000 pull"
  },
  {
    "startMs": 822240,
    "text": "requests to production, which is a wild"
  },
  {
    "startMs": 824959,
    "text": "number. And our pace of releasing the"
  },
  {
    "startMs": 828000,
    "text": "client has also increased. And we cut a"
  },
  {
    "startMs": 830399,
    "text": "new stable version about once a day."
  },
  {
    "startMs": 834079,
    "text": "And then the core team is also shipping"
  },
  {
    "startMs": 836079,
    "text": "much more. And we also have uh a lot"
  },
  {
    "startMs": 838639,
    "text": "more casual or drive by contributors who"
  },
  {
    "startMs": 841920,
    "text": "don't always have a lot of context uh"
  },
  {
    "startMs": 844560,
    "text": "but they're also contributing. Um, and"
  },
  {
    "startMs": 846880,
    "text": "in October we had about 227 people who"
  },
  {
    "startMs": 850959,
    "text": "contributed at least one pull request."
  },
  {
    "startMs": 854959,
    "text": "So despite this velocity accelerating,"
  },
  {
    "startMs": 856800,
    "text": "you might be thinking to yourself like,"
  },
  {
    "startMs": 858000,
    "text": "isn't this going to be chaos? You know,"
  },
  {
    "startMs": 859440,
    "text": "isn't this going to be a bunch of flop?"
  },
  {
    "startMs": 861360,
    "text": "But actually, the number of uh PRs that"
  },
  {
    "startMs": 863680,
    "text": "get reverted has actually gotten lower"
  },
  {
    "startMs": 865279,
    "text": "on a relative basis and it's steady at"
  },
  {
    "startMs": 867440,
    "text": "about half a percent."
  },
  {
    "startMs": 870320,
    "text": "But I think the really interesting thing"
  },
  {
    "startMs": 872160,
    "text": "about this shipping pace and this volume"
  },
  {
    "startMs": 874639,
    "text": "of pull requests is that the performance"
  },
  {
    "startMs": 878079,
    "text": "of Grabbot has stayed not just stable"
  },
  {
    "startMs": 880800,
    "text": "but has improved on multiple dimensions"
  },
  {
    "startMs": 882959,
    "text": "compared to when we launched. So startup"
  },
  {
    "startMs": 885920,
    "text": "typing, opening new chats, switching"
  },
  {
    "startMs": 888160,
    "text": "chats, all of this has gotten faster and"
  },
  {
    "startMs": 891279,
    "text": "keep in mind Grothbot is still an"
  },
  {
    "startMs": 893839,
    "text": "Electron app for now. Uh so there is a"
  },
  {
    "startMs": 896720,
    "text": "ceiling, right? But it's already much"
  },
  {
    "startMs": 898959,
    "text": "faster than many other agent UIs out"
  },
  {
    "startMs": 901279,
    "text": "there. And for an Electron app with this"
  },
  {
    "startMs": 905199,
    "text": "high velocity and this many people"
  },
  {
    "startMs": 907279,
    "text": "contributing, I'd say that's pretty"
  },
  {
    "startMs": 909279,
    "text": "good."
  },
  {
    "startMs": 911360,
    "text": "So before you jump into building out a"
  },
  {
    "startMs": 913360,
    "text": "software factory and you know automating"
  },
  {
    "startMs": 915360,
    "text": "everything that you can, whenever you"
  },
  {
    "startMs": 917600,
    "text": "correct your agent, try and do it by"
  },
  {
    "startMs": 919440,
    "text": "improving your codebase first. you know,"
  },
  {
    "startMs": 921440,
    "text": "start with a lint rule to prevent new"
  },
  {
    "startMs": 923199,
    "text": "occurrences of anti-atterns and then"
  },
  {
    "startMs": 925519,
    "text": "clean up those uh old patterns as well"
  },
  {
    "startMs": 928079,
    "text": "and then make sure that your"
  },
  {
    "startMs": 929440,
    "text": "verification story is really strong."
  },
  {
    "startMs": 932880,
    "text": "A lot of this work is a prerequisite to"
  },
  {
    "startMs": 935040,
    "text": "automation anyway. And if your inner"
  },
  {
    "startMs": 937600,
    "text": "coding loop uh produces slop, then"
  },
  {
    "startMs": 940639,
    "text": "automating that loop just produces more"
  },
  {
    "startMs": 942720,
    "text": "slop and you need to throw more and more"
  },
  {
    "startMs": 944639,
    "text": "and more tokens at the problem, which we"
  },
  {
    "startMs": 946560,
    "text": "all know is very expensive."
  },
  {
    "startMs": 949440,
    "text": "And once you've made these investments"
  },
  {
    "startMs": 950959,
    "text": "into you know code into your codebase"
  },
  {
    "startMs": 953199,
    "text": "static analysis runtime informal"
  },
  {
    "startMs": 955279,
    "text": "verification you'll start to find that"
  },
  {
    "startMs": 957920,
    "text": "many things become possible with these"
  },
  {
    "startMs": 960160,
    "text": "foundations and that you can really"
  },
  {
    "startMs": 962320,
    "text": "easily build your own factory on top of"
  },
  {
    "startMs": 964639,
    "text": "these foundations"
  },
  {
    "startMs": 967040,
    "text": "and when you're ready is a great way to"
  },
  {
    "startMs": 969440,
    "text": "build that uh outer loop. We have tons"
  },
  {
    "startMs": 971600,
    "text": "of connectors, routines that can kick"
  },
  {
    "startMs": 973360,
    "text": "off cursor cloud agents and projects uh"
  },
  {
    "startMs": 976560,
    "text": "which let you orchestrate thousands of u"
  },
  {
    "startMs": 978720,
    "text": "hundreds or even thousands of agents."
  },
  {
    "startMs": 981040,
    "text": "And as a quick example, you know,"
  },
  {
    "startMs": 982800,
    "text": "whenever we cut a new release now,"
  },
  {
    "startMs": 984560,
    "text": "Grockbot will automatically spin up a"
  },
  {
    "startMs": 986639,
    "text": "new cursor project for us that will fuzz"
  },
  {
    "startMs": 988959,
    "text": "the entire application, the entire"
  },
  {
    "startMs": 990800,
    "text": "build, and it will find bugs uh by"
  },
  {
    "startMs": 993199,
    "text": "running the app and using it like a real"
  },
  {
    "startMs": 994800,
    "text": "user. And then it's going to open fixed"
  },
  {
    "startMs": 997600,
    "text": "PRs for each of those bugs that it finds"
  },
  {
    "startMs": 999680,
    "text": "and it's going to automerge it as uh"
  },
  {
    "startMs": 1002720,
    "text": "crazy as that sounds. And that's because"
  },
  {
    "startMs": 1004800,
    "text": "our code base is good enough that pull"
  },
  {
    "startMs": 1006959,
    "text": "requests tend to just be good by"
  },
  {
    "startMs": 1008639,
    "text": "default. And most product code that we"
  },
  {
    "startMs": 1010720,
    "text": "land no longer really requires deep"
  },
  {
    "startMs": 1012800,
    "text": "human review."
  },
  {
    "startMs": 1016160,
    "text": "So when you have solid foundations and"
  },
  {
    "startMs": 1019279,
    "text": "good organic architecture, you can turn"
  },
  {
    "startMs": 1021759,
    "text": "slop into gold and you can ship with"
  },
  {
    "startMs": 1024400,
    "text": "high quality uh high velocity without"
  },
  {
    "startMs": 1027520,
    "text": "compromising on quality. My name is"
  },
  {
    "startMs": 1029919,
    "text": "Potato, also known as Lauren. Thanks you"
  },
  {
    "startMs": 1032079,
    "text": "again for listening to my talk."
  },
  {
    "startMs": 1034256,
    "text": "[applause]"
  }
]
