[
  {
    "startMs": 12400,
    "text": "How's everyone doing?"
  },
  {
    "startMs": 14799,
    "text": "Awesome. Uh, my name is Du. I'm one of"
  },
  {
    "startMs": 16800,
    "text": "the co-founders of a company called"
  },
  {
    "startMs": 18640,
    "text": "Gravile. And at Grapile, we're working"
  },
  {
    "startMs": 21279,
    "text": "on AI agents that validate poll requests"
  },
  {
    "startMs": 25119,
    "text": "with full context of the codebase. The"
  },
  {
    "startMs": 27439,
    "text": "way Graval does this is every time"
  },
  {
    "startMs": 28960,
    "text": "there's a pull request, we have a swarm"
  },
  {
    "startMs": 30400,
    "text": "of agents and they go and look at every"
  },
  {
    "startMs": 32800,
    "text": "file that's changed, every file that's"
  },
  {
    "startMs": 35200,
    "text": "related to the files that changed to"
  },
  {
    "startMs": 36559,
    "text": "figure out if there's bugs. And it also"
  },
  {
    "startMs": 38160,
    "text": "spins up your code in a sandbox,"
  },
  {
    "startMs": 40239,
    "text": "installs the dependencies, spins up"
  },
  {
    "startMs": 42719,
    "text": "local host, clicks around to try to"
  },
  {
    "startMs": 44640,
    "text": "break things, mocks inputs, and"
  },
  {
    "startMs": 47039,
    "text": "everything else to figure out if the"
  },
  {
    "startMs": 48320,
    "text": "code is broken. But I'm going to talk"
  },
  {
    "startMs": 50879,
    "text": "about something a little bit different,"
  },
  {
    "startMs": 52879,
    "text": "which is fully autonomous coding agents."
  },
  {
    "startMs": 56239,
    "text": "So"
  },
  {
    "startMs": 57920,
    "text": "I first moved to San Francisco three"
  },
  {
    "startMs": 59199,
    "text": "years ago to work on AI coding. And this"
  },
  {
    "startMs": 60879,
    "text": "is because GBT 3.5 which came out in"
  },
  {
    "startMs": 63760,
    "text": "2022 was the first model that seemed"
  },
  {
    "startMs": 66080,
    "text": "like it was really good at programming"
  },
  {
    "startMs": 68159,
    "text": "at the time. Code complete tab complete"
  },
  {
    "startMs": 71040,
    "text": "was like the main paradigm for coding."
  },
  {
    "startMs": 73040,
    "text": "The most exciting thing that was"
  },
  {
    "startMs": 74080,
    "text": "happening was you had tab complete from"
  },
  {
    "startMs": 75360,
    "text": "cursor and you had co-pilots code"
  },
  {
    "startMs": 77680,
    "text": "complete and that was like the way in"
  },
  {
    "startMs": 79840,
    "text": "which AI coding works. In 2024, for the"
  },
  {
    "startMs": 83920,
    "text": "first time, multifile editing started to"
  },
  {
    "startMs": 86479,
    "text": "work. Cursor was the first to do this"
  },
  {
    "startMs": 88159,
    "text": "and then a couple of other products"
  },
  {
    "startMs": 89600,
    "text": "followed. But now, for the first time,"
  },
  {
    "startMs": 91439,
    "text": "AI could simultaneously edit multiple"
  },
  {
    "startMs": 93600,
    "text": "files at once. But things got really"
  },
  {
    "startMs": 96079,
    "text": "interesting in 2025 because for the"
  },
  {
    "startMs": 97520,
    "text": "first time, we had autonomous agents."
  },
  {
    "startMs": 99360,
    "text": "They could be given a task and they"
  },
  {
    "startMs": 100799,
    "text": "could go and fulfill the task and create"
  },
  {
    "startMs": 102320,
    "text": "entire pull requests all at once. And"
  },
  {
    "startMs": 103920,
    "text": "this came to a precipice in December of"
  },
  {
    "startMs": 105680,
    "text": "last year when coding agents became"
  },
  {
    "startMs": 108399,
    "text": "literally completely autonomous. And"
  },
  {
    "startMs": 110000,
    "text": "anyone that's working in AI coding knows"
  },
  {
    "startMs": 111360,
    "text": "that December when the new models came"
  },
  {
    "startMs": 112799,
    "text": "out was sort of this like watershed"
  },
  {
    "startMs": 114560,
    "text": "moment in the history of AI coding"
  },
  {
    "startMs": 116079,
    "text": "because these products for the first"
  },
  {
    "startMs": 117280,
    "text": "time were like truly autonomous in how"
  },
  {
    "startMs": 119119,
    "text": "they programmed. And all over Twitter"
  },
  {
    "startMs": 122399,
    "text": "were all these crazy stories of people"
  },
  {
    "startMs": 124320,
    "text": "that were having these agents spin off"
  },
  {
    "startMs": 125840,
    "text": "and open a 100 pull requests a day. They"
  },
  {
    "startMs": 128080,
    "text": "were experimenting with polyphasic sleep"
  },
  {
    "startMs": 130080,
    "text": "so they could stay awake to re-trigger"
  },
  {
    "startMs": 132400,
    "text": "their agents from time to time. And as"
  },
  {
    "startMs": 134800,
    "text": "someone that started programming before"
  },
  {
    "startMs": 137200,
    "text": "agents, I was very skeptical that real"
  },
  {
    "startMs": 139840,
    "text": "companies could program this way. So I"
  },
  {
    "startMs": 141840,
    "text": "was really really curious, are these"
  },
  {
    "startMs": 143120,
    "text": "fully vi PRs really actually good? Is"
  },
  {
    "startMs": 145760,
    "text": "this something that's sort of a Twitter"
  },
  {
    "startMs": 147200,
    "text": "hype and you can have independent"
  },
  {
    "startMs": 148800,
    "text": "developers and and maybe like small"
  },
  {
    "startMs": 150800,
    "text": "startups with no real customers do this?"
  },
  {
    "startMs": 152480,
    "text": "But anyone with real customers with an"
  },
  {
    "startMs": 155360,
    "text": "actually commercially viable code base,"
  },
  {
    "startMs": 158400,
    "text": "could they still use these end-to-end"
  },
  {
    "startMs": 160480,
    "text": "coding agents?"
  },
  {
    "startMs": 162239,
    "text": "And at Grubile, we're fortunate to work"
  },
  {
    "startMs": 164640,
    "text": "with some very large companies. We work"
  },
  {
    "startMs": 166319,
    "text": "with Nvidia, with Coinbase, with Scale,"
  },
  {
    "startMs": 168239,
    "text": "with Data Dog, with American Express."
  },
  {
    "startMs": 169920,
    "text": "And we got really excited to see if"
  },
  {
    "startMs": 173360,
    "text": "these coding agents were being used"
  },
  {
    "startMs": 175280,
    "text": "really at these large companies and if"
  },
  {
    "startMs": 176959,
    "text": "they were any good at doing real world"
  },
  {
    "startMs": 179360,
    "text": "coding."
  },
  {
    "startMs": 182640,
    "text": "So, as an amateur data scientist, I"
  },
  {
    "startMs": 184800,
    "text": "decided to go and dive into our data. We"
  },
  {
    "startMs": 186560,
    "text": "review more than a million poll requests"
  },
  {
    "startMs": 187920,
    "text": "a month and we tend to get a lot of"
  },
  {
    "startMs": 190080,
    "text": "interesting data on what's good and bad"
  },
  {
    "startMs": 191680,
    "text": "about these poll requests. We do this"
  },
  {
    "startMs": 192720,
    "text": "for thousands and thousands of companies"
  },
  {
    "startMs": 194400,
    "text": "and they're usually enterprises or at"
  },
  {
    "startMs": 196560,
    "text": "least companies with serious products"
  },
  {
    "startMs": 198480,
    "text": "with real customers. And so we figured"
  },
  {
    "startMs": 200480,
    "text": "actually studying this data would yield"
  },
  {
    "startMs": 202000,
    "text": "some really interesting results. Was a"
  },
  {
    "startMs": 204080,
    "text": "surprisingly hard to figure out from all"
  },
  {
    "startMs": 205680,
    "text": "the poll requests that GR was reviewing"
  },
  {
    "startMs": 207680,
    "text": "which of the poll requests was actually"
  },
  {
    "startMs": 209760,
    "text": "generated by AI. Surprisingly difficult"
  },
  {
    "startMs": 211599,
    "text": "to do this. The first thing that I tried"
  },
  {
    "startMs": 214000,
    "text": "was I started looking at the GitHub"
  },
  {
    "startMs": 215840,
    "text": "author field. GitHub every commit has an"
  },
  {
    "startMs": 219040,
    "text": "author field and so went through the"
  },
  {
    "startMs": 221280,
    "text": "authors and turns out that less than 1%"
  },
  {
    "startMs": 223680,
    "text": "of pull requests had codeex or clawed or"
  },
  {
    "startMs": 225920,
    "text": "cursor as the author. But it wasn't"
  },
  {
    "startMs": 228319,
    "text": "intuitive to me that only but 1% of all"
  },
  {
    "startMs": 230239,
    "text": "code was fully AI generated. It just"
  },
  {
    "startMs": 231599,
    "text": "seemed intuitive that that number would"
  },
  {
    "startMs": 232799,
    "text": "be much higher. So I started to look for"
  },
  {
    "startMs": 234560,
    "text": "other signals. Now thankfully Claude and"
  },
  {
    "startMs": 238000,
    "text": "Codex and some other products leave a PR"
  },
  {
    "startMs": 240239,
    "text": "description in the footer. In the PR"
  },
  {
    "startMs": 242000,
    "text": "description, they put something along"
  },
  {
    "startMs": 243200,
    "text": "the lines of co-authored by claude or"
  },
  {
    "startMs": 244799,
    "text": "co-authored by cursor and so on. And"
  },
  {
    "startMs": 246640,
    "text": "this gave me a little bit more signal, a"
  },
  {
    "startMs": 248000,
    "text": "little bit more data on which PRs were"
  },
  {
    "startMs": 249920,
    "text": "ostensibly largely, if not completely AI"
  },
  {
    "startMs": 252159,
    "text": "generated. The third thing I looked at"
  },
  {
    "startMs": 254480,
    "text": "was branch name prefixes. If you use"
  },
  {
    "startMs": 256000,
    "text": "Codex, you know that Codex names as"
  },
  {
    "startMs": 257919,
    "text": "branches. And it seemed reasonable that"
  },
  {
    "startMs": 259600,
    "text": "if someone was using these coding agents"
  },
  {
    "startMs": 261519,
    "text": "in such a sense that they were writing"
  },
  {
    "startMs": 263040,
    "text": "the branch names or writing the PR"
  },
  {
    "startMs": 264479,
    "text": "descriptions that the PRs were largely"
  },
  {
    "startMs": 266400,
    "text": "AI generated. That seemed like a"
  },
  {
    "startMs": 267520,
    "text": "reasonable assumption to me. So now I"
  },
  {
    "startMs": 269360,
    "text": "had a good set of signals to indicate"
  },
  {
    "startMs": 270880,
    "text": "whether a poll request was in fact"
  },
  {
    "startMs": 272800,
    "text": "completely or largely AI generated or"
  },
  {
    "startMs": 274720,
    "text": "not. And I came to the result that about"
  },
  {
    "startMs": 277280,
    "text": "a quarter of all the poll requests that"
  },
  {
    "startMs": 278639,
    "text": "Graal was reviewing in any given month"
  },
  {
    "startMs": 280800,
    "text": "were completely or at least largely"
  },
  {
    "startMs": 283199,
    "text": "generated by AI."
  },
  {
    "startMs": 286560,
    "text": "Then I backtracked this data to the last"
  },
  {
    "startMs": 288160,
    "text": "12 months and it turns out that this"
  },
  {
    "startMs": 289840,
    "text": "number is growing really fast. In fact,"
  },
  {
    "startMs": 292320,
    "text": "early last year, fewer than 1% of pull"
  },
  {
    "startMs": 294639,
    "text": "requests had any evidence of being"
  },
  {
    "startMs": 295919,
    "text": "completely AI generated. So this number"
  },
  {
    "startMs": 297680,
    "text": "is going very very fast as model"
  },
  {
    "startMs": 299360,
    "text": "performance is getting better. What's"
  },
  {
    "startMs": 300880,
    "text": "also interesting is that you can't tell"
  },
  {
    "startMs": 303280,
    "text": "when new models came out on this chart."
  },
  {
    "startMs": 305280,
    "text": "It seemed like the progress is very"
  },
  {
    "startMs": 306880,
    "text": "continuous. These things are just"
  },
  {
    "startMs": 308400,
    "text": "generally diffusing into the economy at"
  },
  {
    "startMs": 310400,
    "text": "a generally high pace."
  },
  {
    "startMs": 313120,
    "text": "So then came the next question."
  },
  {
    "startMs": 314960,
    "text": "Everyone's vibe coding. Everyone's"
  },
  {
    "startMs": 316400,
    "text": "producing these PRs that are"
  },
  {
    "startMs": 317440,
    "text": "antiendentic. Not human plus AI but"
  },
  {
    "startMs": 319440,
    "text": "literally AI. The question is are these"
  },
  {
    "startMs": 321120,
    "text": "PRs any good? First question, what does"
  },
  {
    "startMs": 323840,
    "text": "it mean for PR to be good? That seemed"
  },
  {
    "startMs": 325360,
    "text": "like actually a very important question"
  },
  {
    "startMs": 326400,
    "text": "to answer before we got into judging"
  },
  {
    "startMs": 328479,
    "text": "these AI generated PRs. And so I took a"
  },
  {
    "startMs": 331120,
    "text": "few methods to try this. The first one I"
  },
  {
    "startMs": 332800,
    "text": "tried was revert rates. If a pull"
  },
  {
    "startMs": 334880,
    "text": "request reverted, it was probably pretty"
  },
  {
    "startMs": 336639,
    "text": "bad. And so that seemed like a pretty"
  },
  {
    "startMs": 340080,
    "text": "sincere sort of guess at what a bad PR"
  },
  {
    "startMs": 342400,
    "text": "could be. And so I started tracking,"
  },
  {
    "startMs": 344080,
    "text": "okay, which PRs were in fact reverted."
  },
  {
    "startMs": 346320,
    "text": "GitHub conveniently names its branches"
  },
  {
    "startMs": 348320,
    "text": "with a revert dash PR number and PR"
  },
  {
    "startMs": 351039,
    "text": "name. And so I was able to track the"
  },
  {
    "startMs": 352639,
    "text": "rate at which these things were being"
  },
  {
    "startMs": 353919,
    "text": "reverted. And I found some interesting"
  },
  {
    "startMs": 355440,
    "text": "data. Codex PR is reverted about one out"
  },
  {
    "startMs": 358080,
    "text": "of every thousand poll requests. Devon"
  },
  {
    "startMs": 360160,
    "text": "once every three and a half times every"
  },
  {
    "startMs": 362160,
    "text": "thousand poll requests. Humans right in"
  },
  {
    "startMs": 364080,
    "text": "the middle at about two and a half. So"
  },
  {
    "startMs": 365520,
    "text": "there doesn't seem to be very big"
  },
  {
    "startMs": 367039,
    "text": "difference between the rate at which"
  },
  {
    "startMs": 368400,
    "text": "poll requests were reverted from people"
  },
  {
    "startMs": 369919,
    "text": "versus agents in my study. Now I was"
  },
  {
    "startMs": 372639,
    "text": "very skeptical of this and I figured"
  },
  {
    "startMs": 374000,
    "text": "okay this is probably because humans are"
  },
  {
    "startMs": 376080,
    "text": "making agents do the easier work the"
  },
  {
    "startMs": 377840,
    "text": "simpler well scoped pull requests and"
  },
  {
    "startMs": 380080,
    "text": "the more complex work was being done by"
  },
  {
    "startMs": 381759,
    "text": "humans and of course complex work had a"
  },
  {
    "startMs": 383199,
    "text": "higher propensity to be reverted because"
  },
  {
    "startMs": 384720,
    "text": "the poll requests were more complicated"
  },
  {
    "startMs": 386000,
    "text": "had larger surface areas risks and so"
  },
  {
    "startMs": 388000,
    "text": "on. So I decided to measure the average"
  },
  {
    "startMs": 390560,
    "text": "size of PR and whether the revert rates"
  },
  {
    "startMs": 393039,
    "text": "were sort of aligned for both of those"
  },
  {
    "startMs": 395360,
    "text": "and I found really interesting data."
  },
  {
    "startMs": 396880,
    "text": "Turns out there's actually very little"
  },
  {
    "startMs": 398319,
    "text": "correlation, if any, between the size of"
  },
  {
    "startMs": 400319,
    "text": "PRs of humans were getting reverted"
  },
  {
    "startMs": 401759,
    "text": "versus size of PRs from agents that were"
  },
  {
    "startMs": 403520,
    "text": "being reverted. So this does not seem"
  },
  {
    "startMs": 405520,
    "text": "like actually very strong evidence that"
  },
  {
    "startMs": 407840,
    "text": "the human PRs are any better than the"
  },
  {
    "startMs": 409759,
    "text": "agent PRs."
  },
  {
    "startMs": 412000,
    "text": "The second set of signals I started"
  },
  {
    "startMs": 413440,
    "text": "looking at was Graile comments. Now, now"
  },
  {
    "startMs": 414960,
    "text": "Grapple is reviewing all these poll"
  },
  {
    "startMs": 416160,
    "text": "requests and Graile finds P 0's, P1's,"
  },
  {
    "startMs": 419280,
    "text": "P2s across all these changes and it"
  },
  {
    "startMs": 420960,
    "text": "seemed reasonable that if Grapile was"
  },
  {
    "startMs": 423360,
    "text": "finding more bugs in the code that the"
  },
  {
    "startMs": 425199,
    "text": "code was most likely worse. So then I"
  },
  {
    "startMs": 427680,
    "text": "started tracking the counts of P 0, P1's"
  },
  {
    "startMs": 430319,
    "text": "and P2s in all these changes. And"
  },
  {
    "startMs": 432960,
    "text": "interestingly, once again, there was not"
  },
  {
    "startMs": 434639,
    "text": "that much of a difference. In fact,"
  },
  {
    "startMs": 436560,
    "text": "three out of the four agents that we"
  },
  {
    "startMs": 437919,
    "text": "tested performed better than humans in"
  },
  {
    "startMs": 440319,
    "text": "terms of the rate at which they were"
  },
  {
    "startMs": 441360,
    "text": "producing P 0. they were producing fewer"
  },
  {
    "startMs": 443280,
    "text": "P 0 than humans were. Similarly the case"
  },
  {
    "startMs": 446000,
    "text": "with P1's and P2s. Broadly speaking,"
  },
  {
    "startMs": 449680,
    "text": "human generated PRs were about equal in"
  },
  {
    "startMs": 452000,
    "text": "quality to agent generated PRs based on"
  },
  {
    "startMs": 453840,
    "text": "this data."
  },
  {
    "startMs": 463840,
    "text": "have the agent go and look at those"
  },
  {
    "startMs": 466000,
    "text": "comments, address them, and make a new"
  },
  {
    "startMs": 467840,
    "text": "commit on the pull request branch. That"
  },
  {
    "startMs": 469680,
    "text": "is the most common way to use guptile."
  },
  {
    "startMs": 471360,
    "text": "And so it seems reasonable that if the"
  },
  {
    "startMs": 472960,
    "text": "pull requests were higher quality, it"
  },
  {
    "startMs": 474720,
    "text": "would require fewer iterations before"
  },
  {
    "startMs": 476319,
    "text": "they would be merged. And so I started"
  },
  {
    "startMs": 477840,
    "text": "tracking the number of iterations, the"
  },
  {
    "startMs": 479280,
    "text": "number of review rounds between when the"
  },
  {
    "startMs": 481039,
    "text": "poll request was opened and when it was"
  },
  {
    "startMs": 482720,
    "text": "merged."
  },
  {
    "startMs": 484639,
    "text": "Sure enough, very little difference."
  },
  {
    "startMs": 486879,
    "text": "Devon's PRs 2.1, Codex PR is 2.45, four,"
  },
  {
    "startMs": 490479,
    "text": "five number of review cycles to merge"
  },
  {
    "startMs": 492960,
    "text": "and humans right in the middle. Once"
  },
  {
    "startMs": 494879,
    "text": "again, very little if any statistical"
  },
  {
    "startMs": 497039,
    "text": "difference between human generated and"
  },
  {
    "startMs": 498639,
    "text": "AI generated pull request in terms of"
  },
  {
    "startMs": 500319,
    "text": "how many iterations before they were"
  },
  {
    "startMs": 501759,
    "text": "ready to merge."
  },
  {
    "startMs": 504400,
    "text": "So now that there wasn't any strong"
  },
  {
    "startMs": 506080,
    "text": "evidence that agent generated PRs were"
  },
  {
    "startMs": 508400,
    "text": "worse than human generated PRs, I got"
  },
  {
    "startMs": 510000,
    "text": "curious if there were qualitative"
  },
  {
    "startMs": 511120,
    "text": "differences. Maybe the types of ways in"
  },
  {
    "startMs": 513039,
    "text": "which agents failed were different from"
  },
  {
    "startMs": 514719,
    "text": "the types of ways that humans failed."
  },
  {
    "startMs": 516880,
    "text": "And so then I looked at the corpus of"
  },
  {
    "startMs": 518800,
    "text": "reptiles comments. It makes on average"
  },
  {
    "startMs": 520800,
    "text": "about four comments per pull request. So"
  },
  {
    "startMs": 522479,
    "text": "we had this corpus of several million"
  },
  {
    "startMs": 524080,
    "text": "comments across the last several months."
  },
  {
    "startMs": 526160,
    "text": "And so I started scanning them for"
  },
  {
    "startMs": 527600,
    "text": "specific phrases and words. For"
  },
  {
    "startMs": 529760,
    "text": "instance, SQL injection or N plus1"
  },
  {
    "startMs": 532560,
    "text": "query. And I plotted the frequency with"
  },
  {
    "startMs": 534399,
    "text": "which these terms occur in graphile"
  },
  {
    "startMs": 535600,
    "text": "comments for these various agents."
  },
  {
    "startMs": 538959,
    "text": "And so I made this chart of patterns of"
  },
  {
    "startMs": 541040,
    "text": "failure."
  },
  {
    "startMs": 543279,
    "text": "To interpret this chart, you can assume"
  },
  {
    "startMs": 545040,
    "text": "that 1x is the human propensity for"
  },
  {
    "startMs": 547200,
    "text": "producing that type of error across the"
  },
  {
    "startMs": 548720,
    "text": "entire chart. And you can kind of see"
  },
  {
    "startMs": 550480,
    "text": "that there's actually quite a lot of"
  },
  {
    "startMs": 552000,
    "text": "variation in the types of failures that"
  },
  {
    "startMs": 554320,
    "text": "these agents seem to produce. For"
  },
  {
    "startMs": 556480,
    "text": "instance, Claude is one and a half times"
  },
  {
    "startMs": 558320,
    "text": "more likely to produce a SQL injection"
  },
  {
    "startMs": 560480,
    "text": "error than humans. Devon is about half"
  },
  {
    "startMs": 563120,
    "text": "as likely as humans to produce a off"
  },
  {
    "startMs": 566480,
    "text": "bypass issue."
  },
  {
    "startMs": 569120,
    "text": "I found it very interesting that there"
  },
  {
    "startMs": 570399,
    "text": "was this much variation in how these"
  },
  {
    "startMs": 571519,
    "text": "agents were performing and how different"
  },
  {
    "startMs": 574080,
    "text": "their failure modes were from humans."
  },
  {
    "startMs": 578080,
    "text": "So it turns out that in spite of my"
  },
  {
    "startMs": 580640,
    "text": "initial skepticism around the enterprise"
  },
  {
    "startMs": 583680,
    "text": "usability of endto-end coding agents,"
  },
  {
    "startMs": 586480,
    "text": "the evidence seems to suggest that"
  },
  {
    "startMs": 587760,
    "text": "they're here and they probably can"
  },
  {
    "startMs": 589279,
    "text": "contribute in real meaningful ways to"
  },
  {
    "startMs": 591360,
    "text": "enterprise coding environments. And so"
  },
  {
    "startMs": 594160,
    "text": "we started to think a little bit more"
  },
  {
    "startMs": 595279,
    "text": "about what code review would look like"
  },
  {
    "startMs": 596959,
    "text": "in such a world. Here's an interesting"
  },
  {
    "startMs": 599040,
    "text": "stat. Today, Graal is used by several"
  },
  {
    "startMs": 601839,
    "text": "tens of thousands of engineers every"
  },
  {
    "startMs": 603839,
    "text": "single week to review all of their code."
  },
  {
    "startMs": 605839,
    "text": "So, we also know generally speaking the"
  },
  {
    "startMs": 608080,
    "text": "number of pull requests that each of"
  },
  {
    "startMs": 609600,
    "text": "these people are writing. The median"
  },
  {
    "startMs": 611839,
    "text": "grubile user writes 50 pull requests a"
  },
  {
    "startMs": 614160,
    "text": "month. So, call it about two plus per"
  },
  {
    "startMs": 616399,
    "text": "workday. The 90th percentile writes 500"
  },
  {
    "startMs": 619360,
    "text": "poll requests per month. That is a"
  },
  {
    "startMs": 621279,
    "text": "drastic difference between the median"
  },
  {
    "startMs": 622800,
    "text": "and the P90. The P99 is in the thousands"
  },
  {
    "startMs": 625839,
    "text": "of pull requests. That is very"
  },
  {
    "startMs": 627760,
    "text": "interesting in its own sense because"
  },
  {
    "startMs": 628959,
    "text": "that means that the people on the"
  },
  {
    "startMs": 630079,
    "text": "margins are actually producing poll"
  },
  {
    "startMs": 631519,
    "text": "requests at the rate at which they're"
  },
  {
    "startMs": 632640,
    "text": "coming up with new ideas. And beyond"
  },
  {
    "startMs": 635920,
    "text": "being interesting, it also makes you"
  },
  {
    "startMs": 637040,
    "text": "wonder well there existing systems for"
  },
  {
    "startMs": 638560,
    "text": "validating this code which is manual"
  },
  {
    "startMs": 640880,
    "text": "code review of course testing maybe you"
  },
  {
    "startMs": 643440,
    "text": "have a QA firm that you work with"
  },
  {
    "startMs": 644959,
    "text": "naturally can scale to that same degree."
  },
  {
    "startMs": 647120,
    "text": "And at Grav, we decided to take sort of"
  },
  {
    "startMs": 648720,
    "text": "a first principles view at what really"
  },
  {
    "startMs": 650399,
    "text": "good validation could look like. Instead"
  },
  {
    "startMs": 652240,
    "text": "of saying that we wanted to automate QA"
  },
  {
    "startMs": 653839,
    "text": "or automate testing or automate code"
  },
  {
    "startMs": 655680,
    "text": "review, we took a step back and said,"
  },
  {
    "startMs": 657600,
    "text": "what would we need to do for anyone to"
  },
  {
    "startMs": 659680,
    "text": "be able to merge hundreds of pull"
  },
  {
    "startMs": 661200,
    "text": "requests a month in an enterprise"
  },
  {
    "startMs": 662720,
    "text": "environment where it really matters that"
  },
  {
    "startMs": 664399,
    "text": "the code is correct? What would need to"
  },
  {
    "startMs": 666000,
    "text": "happen between when the code was"
  },
  {
    "startMs": 667360,
    "text": "expressed into a pull request and when"
  },
  {
    "startMs": 669200,
    "text": "it was merged and deployed safely? We"
  },
  {
    "startMs": 671440,
    "text": "figured we only actually had to answer"
  },
  {
    "startMs": 673200,
    "text": "three questions. The first one is does"
  },
  {
    "startMs": 675360,
    "text": "this change violate the user contract?"
  },
  {
    "startMs": 677519,
    "text": "The second one, does it increase the"
  },
  {
    "startMs": 679120,
    "text": "propensity of a future violation of the"
  },
  {
    "startMs": 681360,
    "text": "user contract, whatever the user"
  },
  {
    "startMs": 682880,
    "text": "contract might be for that application?"
  },
  {
    "startMs": 684880,
    "text": "And third, does it fulfill the intent"
  },
  {
    "startMs": 686640,
    "text": "that the author described? Did the poll"
  },
  {
    "startMs": 688160,
    "text": "request do the thing that the author"
  },
  {
    "startMs": 689519,
    "text": "wanted to do? And so we started"
  },
  {
    "startMs": 692240,
    "text": "approaching this problem from base"
  },
  {
    "startMs": 694640,
    "text": "ground level and said, okay, agents can"
  },
  {
    "startMs": 696880,
    "text": "probably figure out if something's going"
  },
  {
    "startMs": 698720,
    "text": "to violate the user contract and detect"
  },
  {
    "startMs": 700240,
    "text": "bugs. If you let it spin up the code in"
  },
  {
    "startMs": 702720,
    "text": "a sandbox, have it install the"
  },
  {
    "startMs": 704560,
    "text": "dependencies, mock the inputs, run the"
  },
  {
    "startMs": 706320,
    "text": "browser agents, you can probably start"
  },
  {
    "startMs": 707680,
    "text": "to discover most of the issues that"
  },
  {
    "startMs": 710160,
    "text": "could occur, you get a pretty high"
  },
  {
    "startMs": 711839,
    "text": "degree of confidence on merge. Today,"
  },
  {
    "startMs": 714000,
    "text": "almost a fifth of all the poll request"
  },
  {
    "startMs": 715760,
    "text": "or gravile reviews are merged without"
  },
  {
    "startMs": 717279,
    "text": "any human review or without any human"
  },
  {
    "startMs": 718640,
    "text": "testing, which I find very interesting"
  },
  {
    "startMs": 720320,
    "text": "and that is a number that we care a lot"
  },
  {
    "startMs": 721600,
    "text": "about and want to bring higher and"
  },
  {
    "startMs": 723279,
    "text": "higher, of course, within the guard"
  },
  {
    "startMs": 724959,
    "text": "rails of producing really high quality"
  },
  {
    "startMs": 726720,
    "text": "code."
  },
  {
    "startMs": 730000,
    "text": "Thank you so much. My name is D, one of"
  },
  {
    "startMs": 732320,
    "text": "the co-founders of Graptile. We have a"
  },
  {
    "startMs": 733920,
    "text": "booth here which you should come to. And"
  },
  {
    "startMs": 735519,
    "text": "if you're interested in trying Graptile,"
  },
  {
    "startMs": 737440,
    "text": "you can find us at gretile.com and try"
  },
  {
    "startMs": 739760,
    "text": "it today for free. And we'd love to hear"
  },
  {
    "startMs": 742000,
    "text": "your feedback. Thank you so much."
  }
]