1 01:01:02,400 --> 01:01:04,390 Hello everyone. 2 01:01:04,400 --> 01:01:06,390 Don't know if you can see my slides, but 3 01:01:06,400 --> 01:01:09,030 uh my name is Potato, also known as 4 01:01:09,040 --> 01:01:11,430 Lauren, and I'm very excited to chat 5 01:01:11,440 --> 01:01:13,589 with all of you today. Maybe you can get 6 01:01:13,599 --> 01:01:15,430 a big round of applause for our tech 7 01:01:15,440 --> 01:01:17,857 team helping me with my laptop. 8 01:01:17,867 --> 01:01:24,630 [applause] 9 01:01:24,640 --> 01:01:27,349 Good. 10 01:01:27,359 --> 01:01:34,630 I don't see it. 11 01:01:34,640 --> 01:01:37,030 >> Ah, there you go. 12 01:01:37,040 --> 01:01:40,390 All right. Um, so again, I'm my name is 13 01:01:40,400 --> 01:01:43,670 Potato. Uh, I work on Grabbot at 14 01:01:43,680 --> 01:01:45,190 SpaceXsi. 15 01:01:45,200 --> 01:01:46,950 And can I just get a quick show of 16 01:01:46,960 --> 01:01:49,510 hands? How many of you have ever looked 17 01:01:49,520 --> 01:01:51,990 at code that somebody else wrote and you 18 01:01:52,000 --> 01:01:54,537 thought that's all slap? 19 01:01:54,547 --> 01:01:55,750 >> [laughter] 20 01:01:55,760 --> 01:01:59,270 >> Well, I have too, 21 01:01:59,280 --> 01:02:00,789 but today I'm going to convince you that 22 01:02:00,799 --> 01:02:06,549 you can turn that slop into gold. 23 01:02:06,559 --> 01:02:08,549 Uh, and so I did something crazy last 24 01:02:08,559 --> 01:02:11,270 month and I shipped 3,000 ploy crust to 25 01:02:11,280 --> 01:02:14,069 production. Now, before agents, we never 26 01:02:14,079 --> 01:02:16,309 really cared about volume. Uh, except 27 01:02:16,319 --> 01:02:18,549 for the outliers. 28 01:02:18,559 --> 01:02:20,549 And at my previous company, we called 29 01:02:20,559 --> 01:02:22,549 these people coding machines. people 30 01:02:22,559 --> 01:02:24,789 like Brett Taylor who famously rewrote 31 01:02:24,799 --> 01:02:27,430 all of Google Maps uh in the weekend 32 01:02:27,440 --> 01:02:30,470 just an insanely productive engineer. Uh 33 01:02:30,480 --> 01:02:32,950 and well with agents anyone can now be a 34 01:02:32,960 --> 01:02:35,430 coding machine. And my thesis is that 35 01:02:35,440 --> 01:02:37,750 the job of a software engineer the meta 36 01:02:37,760 --> 01:02:39,750 of being a software engineer is figuring 37 01:02:39,760 --> 01:02:42,549 out how to maintain or even improve the 38 01:02:42,559 --> 01:02:44,710 quality of your code while massively 39 01:02:44,720 --> 01:02:47,670 increasing your productivity. 40 01:02:47,680 --> 01:02:49,190 To get there, some people like to call 41 01:02:49,200 --> 01:02:51,829 this a software factory. Uh, but I like 42 01:02:51,839 --> 01:02:54,309 to keep it simple and I think of it as 43 01:02:54,319 --> 01:02:56,950 all about trust. 44 01:02:56,960 --> 01:02:59,349 Not blind trust, but investing in your 45 01:02:59,359 --> 01:03:01,510 agents environment so you can trust that 46 01:03:01,520 --> 01:03:03,190 your agents are going to do the right 47 01:03:03,200 --> 01:03:06,630 thing. Now, it takes quite a lot of 48 01:03:06,640 --> 01:03:09,270 effort to get there. Um, and it took me 49 01:03:09,280 --> 01:03:11,510 six months to get to a point where I 50 01:03:11,520 --> 01:03:13,750 could ship thousands of pull requests, a 51 01:03:13,760 --> 01:03:15,910 lot of them in my sleep, but it's 52 01:03:15,920 --> 01:03:18,230 certainly possible and it will take you 53 01:03:18,240 --> 01:03:20,789 much less time than I did. So let's 54 01:03:20,799 --> 01:03:22,710 start by talking about how we move from 55 01:03:22,720 --> 01:03:25,029 a low trust environment to a high trust 56 01:03:25,039 --> 01:03:27,349 one. 57 01:03:27,359 --> 01:03:29,349 So this inverse pyramid is how I think 58 01:03:29,359 --> 01:03:31,990 about where to spend time on your agents 59 01:03:32,000 --> 01:03:33,990 environment. 60 01:03:34,000 --> 01:03:36,470 The first two layers are deterministic. 61 01:03:36,480 --> 01:03:38,789 So using your code itself. So things 62 01:03:38,799 --> 01:03:41,430 like good API design, architecture, 63 01:03:41,440 --> 01:03:44,390 refactoring or using lint rules, type 64 01:03:44,400 --> 01:03:46,870 systems, compilers or you know different 65 01:03:46,880 --> 01:03:49,430 kinds of testing. And you can use both 66 01:03:49,440 --> 01:03:51,510 of these layers together to constrain 67 01:03:51,520 --> 01:03:53,990 what agents can do and force them to do 68 01:03:54,000 --> 01:03:56,870 the right thing by default. 69 01:03:56,880 --> 01:03:59,190 Verification, both runtime and formal, 70 01:03:59,200 --> 01:04:02,150 are great tools to layer on to find bugs 71 01:04:02,160 --> 01:04:04,230 and check properties and the invariance 72 01:04:04,240 --> 01:04:05,750 in your system that you want to hold 73 01:04:05,760 --> 01:04:08,789 true and also help you check whether or 74 01:04:08,799 --> 01:04:10,870 not the agent understood your goals and 75 01:04:10,880 --> 01:04:12,950 your intent. 76 01:04:12,960 --> 01:04:14,789 And then finally, you want to layer with 77 01:04:14,799 --> 01:04:16,789 a little bit of pros. So things like 78 01:04:16,799 --> 01:04:20,549 rules, skills, agents.mmd, things that 79 01:04:20,559 --> 01:04:23,349 require judgment. 80 01:04:23,359 --> 01:04:25,190 But let's start with your codebase. 81 01:04:25,200 --> 01:04:27,589 Agents generate their outputs based on 82 01:04:27,599 --> 01:04:30,230 their inputs. And so the the quality of 83 01:04:30,240 --> 01:04:32,150 the input really matters. And it just so 84 01:04:32,160 --> 01:04:34,630 happens that your codebase is one of the 85 01:04:34,640 --> 01:04:37,430 largest inputs. So whenever you're 86 01:04:37,440 --> 01:04:39,190 correcting your agent, think about how 87 01:04:39,200 --> 01:04:41,430 you can make that a deterministic check 88 01:04:41,440 --> 01:04:43,829 instead of adding another line to your 89 01:04:43,839 --> 01:04:46,230 rules or your agents.mmd because it's 90 01:04:46,240 --> 01:04:48,390 far more reproducible when it's in code 91 01:04:48,400 --> 01:04:51,430 and it lets you trust it more. 92 01:04:51,440 --> 01:04:53,430 And the analogy that I like to use is 93 01:04:53,440 --> 01:04:55,589 that your codebase is kind of like a 94 01:04:55,599 --> 01:04:58,230 garden. You know, code used to arrive a 95 01:04:58,240 --> 01:05:01,349 few pull requests at a time. And uh I 96 01:05:01,359 --> 01:05:04,390 remember going into work in 2024 and 97 01:05:04,400 --> 01:05:06,150 reviewing the three pull requests that 98 01:05:06,160 --> 01:05:08,470 my colleagues wrote colleagues wrote and 99 01:05:08,480 --> 01:05:12,390 today that number is more like 300. So 100 01:05:12,400 --> 01:05:14,470 uh you know every engineer is now a team 101 01:05:14,480 --> 01:05:17,029 of hundreds or even thousands of agents 102 01:05:17,039 --> 01:05:20,309 and code is arriving so rapidly that it 103 01:05:20,319 --> 01:05:22,950 almost feels like a continuous stream 104 01:05:22,960 --> 01:05:25,589 and it almost feels organic like a 105 01:05:25,599 --> 01:05:27,430 living thing. 106 01:05:27,440 --> 01:05:30,390 And in this analogy, your codebase, the 107 01:05:30,400 --> 01:05:32,309 constraints that you impose, the 108 01:05:32,319 --> 01:05:34,390 guardrails that you put up are the 109 01:05:34,400 --> 01:05:36,870 concrete foundations that the new code 110 01:05:36,880 --> 01:05:39,589 that your agents write grow around just 111 01:05:39,599 --> 01:05:42,069 like uh the building in this picture. 112 01:05:42,079 --> 01:05:43,829 And so in many ways, the job of being a 113 01:05:43,839 --> 01:05:45,829 software engineer is kind of becoming 114 01:05:45,839 --> 01:05:48,870 like gardening 115 01:05:48,880 --> 01:05:50,630 because code is the biggest input for 116 01:05:50,640 --> 01:05:54,150 agents. It means that any workarounds um 117 01:05:54,160 --> 01:05:57,990 anti-atterns deprecated code tech is now 118 01:05:58,000 --> 01:05:59,910 likely to be just copied by the next 119 01:05:59,920 --> 01:06:01,589 agent since it looks like the 120 01:06:01,599 --> 01:06:04,950 established pattern in your codebase and 121 01:06:04,960 --> 01:06:07,430 uh it's like you have some weeds or 122 01:06:07,440 --> 01:06:09,510 pests growing in your garden and then 123 01:06:09,520 --> 01:06:11,190 they spread everywhere before you know 124 01:06:11,200 --> 01:06:15,109 it. 125 01:06:15,119 --> 01:06:17,190 So you want to make your codebase as 126 01:06:17,200 --> 01:06:19,589 pristine as possible so that agents are 127 01:06:19,599 --> 01:06:22,230 always copying the good stuff. All 128 01:06:22,240 --> 01:06:24,630 right, let's look at some real examples. 129 01:06:24,640 --> 01:06:27,029 Uh so Dune is a client framework that we 130 01:06:27,039 --> 01:06:29,190 created for graphbot to implement this 131 01:06:29,200 --> 01:06:32,069 idea for organic architecture. 132 01:06:32,079 --> 01:06:33,349 It's not really that interesting 133 01:06:33,359 --> 01:06:35,430 technically. Uh you can kind of think of 134 01:06:35,440 --> 01:06:38,390 Dune as um kind of like Nex.js for 135 01:06:38,400 --> 01:06:40,630 Electron apps. But the interesting part 136 01:06:40,640 --> 01:06:42,150 of Dune is that it has a lot of 137 01:06:42,160 --> 01:06:44,390 constraints and lit rules that we've 138 01:06:44,400 --> 01:06:46,630 encoded uh a lot of the lessons that 139 01:06:46,640 --> 01:06:48,470 we've learned building desktop 140 01:06:48,480 --> 01:06:50,549 applications. 141 01:06:50,559 --> 01:06:52,950 And one such lesson is that anything 142 01:06:52,960 --> 01:06:55,510 that requires nuance to use well often 143 01:06:55,520 --> 01:06:58,390 leads to slop. Uh you know your classic 144 01:06:58,400 --> 01:07:00,870 example is uh you know use effect in 145 01:07:00,880 --> 01:07:03,829 React, right? It has its use cases uh 146 01:07:03,839 --> 01:07:05,910 but it's often misused. You know some 147 01:07:05,920 --> 01:07:07,829 people joke that you need a PhD to use 148 01:07:07,839 --> 01:07:10,950 use effect. Um and in in other words, it 149 01:07:10,960 --> 01:07:13,029 really requires a lot of nuance to use 150 01:07:13,039 --> 01:07:15,910 ball and agents are not good at using 151 01:07:15,920 --> 01:07:19,430 it. Um so our solution has been to just 152 01:07:19,440 --> 01:07:21,910 ban it in product code and it's really 153 01:07:21,920 --> 01:07:23,990 only allowed in some places inside of 154 01:07:24,000 --> 01:07:26,309 Dune APIs 155 01:07:26,319 --> 01:07:28,630 and effectively this problem has now 156 01:07:28,640 --> 01:07:31,029 been categorically eliminated and we 157 01:07:31,039 --> 01:07:32,549 never need to catch that during code 158 01:07:32,559 --> 01:07:34,789 review again. 159 01:07:34,799 --> 01:07:36,390 Another interesting pattern that we use 160 01:07:36,400 --> 01:07:39,109 is context through directory structure. 161 01:07:39,119 --> 01:07:41,510 Uh so one antiattern that we saw with 162 01:07:41,520 --> 01:07:43,990 cursor which is an electron app is that 163 01:07:44,000 --> 01:07:46,870 agents would often import main thread 164 01:07:46,880 --> 01:07:48,630 code into the renderer thread which 165 01:07:48,640 --> 01:07:52,069 would cause long tasks and jank. 166 01:07:52,079 --> 01:07:55,029 So in uh grapot and in dunes we've made 167 01:07:55,039 --> 01:07:57,349 this very very conventional. Uh we have 168 01:07:57,359 --> 01:07:59,270 these two top level directories 169 01:07:59,280 --> 01:08:01,190 literally called electron renderer and 170 01:08:01,200 --> 01:08:04,549 electron main and imports between them 171 01:08:04,559 --> 01:08:06,710 are forbidden or banned by our 172 01:08:06,720 --> 01:08:09,029 dependency graph analyzer and some lint 173 01:08:09,039 --> 01:08:11,589 rules. And again you know these are 174 01:08:11,599 --> 01:08:13,829 simple lint rules to set up but it means 175 01:08:13,839 --> 01:08:15,750 one less thing that we need to review 176 01:08:15,760 --> 01:08:18,470 and think about uh you know during code 177 01:08:18,480 --> 01:08:20,870 review. 178 01:08:20,880 --> 01:08:23,110 And then one more surprising thing that 179 01:08:23,120 --> 01:08:26,229 I've seen is agents using markdown. So 180 01:08:26,239 --> 01:08:28,630 things like documentation, uh, code 181 01:08:28,640 --> 01:08:31,189 comments, rules or skills, and they use 182 01:08:31,199 --> 01:08:33,669 this as a place to write down their 183 01:08:33,679 --> 01:08:35,669 justifications for not doing the right 184 01:08:35,679 --> 01:08:37,430 thing. Or, I don't know if you've ever 185 01:08:37,440 --> 01:08:39,430 seen this, but they'll over rotate on 186 01:08:39,440 --> 01:08:41,510 some small correction that you made, and 187 01:08:41,520 --> 01:08:43,430 they treat it like a hard rule that can 188 01:08:43,440 --> 01:08:46,709 never be broken ever again. 189 01:08:46,719 --> 01:08:49,110 And here's another example from cursor. 190 01:08:49,120 --> 01:08:51,749 So agents really loved using imperative 191 01:08:51,759 --> 01:08:53,749 timers. So things like set timeout, 192 01:08:53,759 --> 01:08:55,669 request animation frame, request idle 193 01:08:55,679 --> 01:08:57,669 call back. And they were using these as 194 01:08:57,679 --> 01:09:00,390 quick fixes because they were really 195 01:09:00,400 --> 01:09:02,390 patching over an architecture clash 196 01:09:02,400 --> 01:09:05,590 between uh VS Code's imperative style 197 01:09:05,600 --> 01:09:08,390 and React's declarative style. 198 01:09:08,400 --> 01:09:10,630 So our choice of architecture actually 199 01:09:10,640 --> 01:09:13,590 made both the client and the agent 200 01:09:13,600 --> 01:09:15,990 performance much worse. And if you've 201 01:09:16,000 --> 01:09:17,189 ever done any performance work in 202 01:09:17,199 --> 01:09:18,470 JavaScript, you'll know that when you 203 01:09:18,480 --> 01:09:20,630 have tons of timers being rapidly 204 01:09:20,640 --> 01:09:23,510 created and destroyed causes a lot of uh 205 01:09:23,520 --> 01:09:25,910 major garbage collection, which causes a 206 01:09:25,920 --> 01:09:28,309 lot of lag and jen. Uh but the worst 207 01:09:28,319 --> 01:09:30,390 part is that agents wrote a lot of code 208 01:09:30,400 --> 01:09:32,709 comments like you can see in red here 209 01:09:32,719 --> 01:09:34,709 explaining why this quick fix was 210 01:09:34,719 --> 01:09:37,990 justified and before you know it, we had 211 01:09:38,000 --> 01:09:40,550 timers spreading everywhere and the you 212 01:09:40,560 --> 01:09:41,990 know the performance of the app really 213 01:09:42,000 --> 01:09:44,630 suffered. 214 01:09:44,640 --> 01:09:47,030 Now in Grockbot uh we learned our lesson 215 01:09:47,040 --> 01:09:49,829 there and we decided to just ban 99% of 216 01:09:49,839 --> 01:09:52,390 comments uh because they go stale 217 01:09:52,400 --> 01:09:54,630 easily. Uh you know humans just really 218 01:09:54,640 --> 01:09:56,310 don't bother reading the comment slop 219 01:09:56,320 --> 01:09:59,750 anyway. And again it gives an agent the 220 01:09:59,760 --> 01:10:02,709 an escape hatch to justify not doing the 221 01:10:02,719 --> 01:10:05,910 right thing because it's hard. Um and 222 01:10:05,920 --> 01:10:07,430 the worst part about comments though is 223 01:10:07,440 --> 01:10:09,910 that they're not enforcable. So what 224 01:10:09,920 --> 01:10:11,990 seems well-intentioned, you know, like 225 01:10:12,000 --> 01:10:14,070 I'm going to tell the the next agent not 226 01:10:14,080 --> 01:10:16,390 to do something, never ends up actually 227 01:10:16,400 --> 01:10:20,070 being anything that you can trust. So in 228 01:10:20,080 --> 01:10:22,950 Grockbot and Dune and in Dune, uh we now 229 01:10:22,960 --> 01:10:25,669 have lint rules that give uh good and 230 01:10:25,679 --> 01:10:27,669 very helpful error messages telling the 231 01:10:27,679 --> 01:10:30,390 agent to fix the root cause, the real 232 01:10:30,400 --> 01:10:32,390 problem, uh rather than just patch 233 01:10:32,400 --> 01:10:34,550 around it. 234 01:10:34,560 --> 01:10:36,229 And so our general principle is that 235 01:10:36,239 --> 01:10:37,750 every time we correct our agents, it 236 01:10:37,760 --> 01:10:39,430 turns into one of these deterministic 237 01:10:39,440 --> 01:10:41,350 checks or we think about how to 238 01:10:41,360 --> 01:10:43,189 categorically eliminate that problem in 239 01:10:43,199 --> 01:10:45,590 the code through better API design, 240 01:10:45,600 --> 01:10:48,470 through better uh architecture. And if 241 01:10:48,480 --> 01:10:50,709 you do this enough times, you'll find 242 01:10:50,719 --> 01:10:53,110 that you have just less things to look 243 01:10:53,120 --> 01:10:55,270 out for in code review. And at some 244 01:10:55,280 --> 01:10:57,030 point, you'll realize there's there's 245 01:10:57,040 --> 01:10:59,030 nothing left to review, and you can just 246 01:10:59,040 --> 01:11:01,830 let agents merge their most of their own 247 01:11:01,840 --> 01:11:04,870 pull requests. 248 01:11:04,880 --> 01:11:07,350 All right. Uh, moving on. Verification 249 01:11:07,360 --> 01:11:09,510 is also a really big part of how we make 250 01:11:09,520 --> 01:11:12,070 our agents more autonomous and also 251 01:11:12,080 --> 01:11:13,830 increase our confidence and trust in 252 01:11:13,840 --> 01:11:16,310 their work. And our code review process 253 01:11:16,320 --> 01:11:18,070 has now really shrunk into looking at 254 01:11:18,080 --> 01:11:20,470 verification artifacts instead. So 255 01:11:20,480 --> 01:11:22,870 things like videos, screenshots, 256 01:11:22,880 --> 01:11:25,990 diagrams, formal verification. 257 01:11:26,000 --> 01:11:27,990 Um, and the most basic example of this 258 01:11:28,000 --> 01:11:30,550 is runtime verification. uh which is 259 01:11:30,560 --> 01:11:32,310 just a fancy way of saying giving your 260 01:11:32,320 --> 01:11:34,630 agent a way to run the code for itself 261 01:11:34,640 --> 01:11:36,630 and use debugging tools that you know 262 01:11:36,640 --> 01:11:38,550 human developers use as well like the 263 01:11:38,560 --> 01:11:41,350 Chrome DevTools protocol. Um in our 264 01:11:41,360 --> 01:11:43,430 runtime verification skills we have this 265 01:11:43,440 --> 01:11:45,830 thing that we call a feature map and 266 01:11:45,840 --> 01:11:48,390 that's basically kind of like a site map 267 01:11:48,400 --> 01:11:49,830 of all of the features that are 268 01:11:49,840 --> 01:11:51,590 available in your application from the 269 01:11:51,600 --> 01:11:54,070 user's point of view. And so with this 270 01:11:54,080 --> 01:11:56,229 agents don't need to guess when you say 271 01:11:56,239 --> 01:11:57,990 you know the thingy in the left sidebar. 272 01:11:58,000 --> 01:12:01,590 It gives you some shared vocabulary 273 01:12:01,600 --> 01:12:03,669 and we also make use of custom CLIs 274 01:12:03,679 --> 01:12:06,149 inside these skills so that uh agents 275 01:12:06,159 --> 01:12:09,110 can reuse the the CLI and the driver 276 01:12:09,120 --> 01:12:10,709 every time which cuts down on token 277 01:12:10,719 --> 01:12:14,709 usage and is also testable. 278 01:12:14,719 --> 01:12:16,550 Um and then recently we've also started 279 01:12:16,560 --> 01:12:19,030 looking at formal verification. Uh so we 280 01:12:19,040 --> 01:12:21,189 now use TA plus to do some model 281 01:12:21,199 --> 01:12:23,830 checking of major user flows such as uh 282 01:12:23,840 --> 01:12:25,669 notifications across multiple devices 283 01:12:25,679 --> 01:12:28,390 and platforms. Uh we use a lean as well 284 01:12:28,400 --> 01:12:30,790 for proving certain algorithms. Uh for 285 01:12:30,800 --> 01:12:32,709 example, we have our own we wrote our 286 01:12:32,719 --> 01:12:34,630 own custom virtualization library and we 287 01:12:34,640 --> 01:12:36,550 use lean there to check that we don't 288 01:12:36,560 --> 01:12:38,709 have you know flickering or jittering 289 01:12:38,719 --> 01:12:40,870 problems. Um and we're still 290 01:12:40,880 --> 01:12:42,149 experimenting with this approach. 291 01:12:42,159 --> 01:12:44,390 Overall it seems very promising. Uh 292 01:12:44,400 --> 01:12:46,310 we've already found quite a few bugs uh 293 01:12:46,320 --> 01:12:49,350 with formal verification. 294 01:12:49,360 --> 01:12:50,790 uh but it's certainly not a silver 295 01:12:50,800 --> 01:12:54,149 bullet and we're treating it as another 296 01:12:54,159 --> 01:12:57,350 tool, another layer in our agents uh 297 01:12:57,360 --> 01:12:59,590 environment. 298 01:12:59,600 --> 01:13:01,750 Uh and then this is a very fast video 299 01:13:01,760 --> 01:13:03,590 but uh another thing that we just 300 01:13:03,600 --> 01:13:05,189 started looking at is property based 301 01:13:05,199 --> 01:13:07,910 testing as well for UIs. Uh so we use 302 01:13:07,920 --> 01:13:10,229 this library called Bombadil. Um and 303 01:13:10,239 --> 01:13:12,470 this has also caught a few UI bugs for 304 01:13:12,480 --> 01:13:15,910 us already. 305 01:13:15,920 --> 01:13:17,669 Okay, so zooming out, I saw this tweet 306 01:13:17,679 --> 01:13:20,470 from Cindy and six months ago she she 307 01:13:20,480 --> 01:13:22,709 said that she was or she was talking 308 01:13:22,719 --> 01:13:25,030 about not being able to trust her agents 309 01:13:25,040 --> 01:13:27,030 because of subtle bugs that were getting 310 01:13:27,040 --> 01:13:29,030 merged. 311 01:13:29,040 --> 01:13:30,950 And really what was missing for her in 312 01:13:30,960 --> 01:13:32,470 being able to trust her agents was the 313 01:13:32,480 --> 01:13:35,110 verification process. And now 6 months 314 01:13:35,120 --> 01:13:37,669 later, she says that 70% of her daily 315 01:13:37,679 --> 01:13:40,310 tokens go into verification. And I'd say 316 01:13:40,320 --> 01:13:42,470 that that tracks with what I see as 317 01:13:42,480 --> 01:13:44,470 well, that most of my token usage goes 318 01:13:44,480 --> 01:13:46,470 to verification. 319 01:13:46,480 --> 01:13:47,910 And if you really look closely at our 320 01:13:47,920 --> 01:13:50,149 process, none of these things here are 321 01:13:50,159 --> 01:13:52,390 new or novel techniques that we've never 322 01:13:52,400 --> 01:13:54,550 had before. 323 01:13:54,560 --> 01:13:56,390 These are just good software engineering 324 01:13:56,400 --> 01:13:58,790 tools, but ones that might have been 325 01:13:58,800 --> 01:14:01,270 prohibitively expensive to use before as 326 01:14:01,280 --> 01:14:03,430 they were hard to understand or write or 327 01:14:03,440 --> 01:14:06,709 maintain. But agents drastically cut 328 01:14:06,719 --> 01:14:08,550 this cost down. And really, we're 329 01:14:08,560 --> 01:14:10,310 entering a golden age of software 330 01:14:10,320 --> 01:14:12,870 testing and verification because agents 331 01:14:12,880 --> 01:14:14,790 let us test everything to a much greater 332 01:14:14,800 --> 01:14:18,950 degree than used to be economical. 333 01:14:18,960 --> 01:14:20,790 And then formal verification can also 334 01:14:20,800 --> 01:14:22,310 help you better reason about how your 335 01:14:22,320 --> 01:14:23,910 product should work and whether your 336 01:14:23,920 --> 01:14:25,590 agents really understand your intent and 337 01:14:25,600 --> 01:14:28,149 goals. Uh I made this new skill called 338 01:14:28,159 --> 01:14:31,350 state chart uh in my plug-in PAC potato 339 01:14:31,360 --> 01:14:33,510 stack uh which helps you visualize state 340 01:14:33,520 --> 01:14:36,229 machines from uh from the user's point 341 01:14:36,239 --> 01:14:38,390 of view. Uh, and it's I find this much 342 01:14:38,400 --> 01:14:40,390 easier to review alongside the videos 343 01:14:40,400 --> 01:14:43,350 and screenshots that we provide. Um, and 344 01:14:43,360 --> 01:14:45,669 so that's been great. Sorry for the 345 01:14:45,679 --> 01:14:48,709 flashbang. Uh, okay. Now, let's look at 346 01:14:48,719 --> 01:14:51,030 what the promised land looks like. What 347 01:14:51,040 --> 01:14:53,830 do you get in return for all of this uh, 348 01:14:53,840 --> 01:14:56,149 you know, investment? 349 01:14:56,159 --> 01:14:58,550 So, on the graphbot team, our velocity 350 01:14:58,560 --> 01:15:00,630 is uh, taking off and rapidly 351 01:15:00,640 --> 01:15:02,950 increasing. And last month, our team 352 01:15:02,960 --> 01:15:06,149 shipped a ridiculous 11,000 pull 353 01:15:06,159 --> 01:15:08,950 requests to production, which is a wild 354 01:15:08,960 --> 01:15:11,990 number. And our pace of releasing the 355 01:15:12,000 --> 01:15:14,310 client has also increased. And we cut a 356 01:15:14,320 --> 01:15:17,990 new stable version about once a day. 357 01:15:18,000 --> 01:15:19,990 And then the core team is also shipping 358 01:15:20,000 --> 01:15:22,550 much more. And we also have uh a lot 359 01:15:22,560 --> 01:15:25,910 more casual or driveby contributors who 360 01:15:25,920 --> 01:15:28,550 don't always have a lot of context uh 361 01:15:28,560 --> 01:15:30,709 but they're also contributing. Um, and 362 01:15:30,719 --> 01:15:34,870 in October we had about 227 people who 363 01:15:34,880 --> 01:15:38,790 contributed at least one pull request. 364 01:15:38,800 --> 01:15:40,630 So despite this velocity accelerating, 365 01:15:40,640 --> 01:15:41,830 you might be thinking to yourself like, 366 01:15:41,840 --> 01:15:43,350 isn't this going to be chaos? You know, 367 01:15:43,360 --> 01:15:45,189 isn't this going to be a bunch of slop? 368 01:15:45,199 --> 01:15:47,510 But actually, the number of uh PRs that 369 01:15:47,520 --> 01:15:49,189 get reverted has actually gotten lower 370 01:15:49,199 --> 01:15:51,350 on a relative basis and is steady at 371 01:15:51,360 --> 01:15:54,310 about half a percent. 372 01:15:54,320 --> 01:15:56,149 But I think the really interesting thing 373 01:15:56,159 --> 01:15:58,630 about this shipping pace and this volume 374 01:15:58,640 --> 01:16:01,990 of pull requests is that the performance 375 01:16:02,000 --> 01:16:04,709 of Grathbot has stayed not just stable 376 01:16:04,719 --> 01:16:06,870 but has improved on multiple dimensions 377 01:16:06,880 --> 01:16:09,910 compared to when we launched. So startup 378 01:16:09,920 --> 01:16:12,149 typing, opening new chats, switching 379 01:16:12,159 --> 01:16:15,270 chats, all of this has gotten faster and 380 01:16:15,280 --> 01:16:17,830 keep in mind Grathbot is still an 381 01:16:17,840 --> 01:16:20,630 Electron app for now. Uh so there is a 382 01:16:20,640 --> 01:16:22,950 ceiling, right? But it's already much 383 01:16:22,960 --> 01:16:25,270 faster than many other agent UIs out 384 01:16:25,280 --> 01:16:29,189 there. And for an Electron app with this 385 01:16:29,199 --> 01:16:31,189 high velocity and this many people 386 01:16:31,199 --> 01:16:33,189 contributing, I'd say that's pretty 387 01:16:33,199 --> 01:16:35,270 good. 388 01:16:35,280 --> 01:16:37,270 So before you jump into building out a 389 01:16:37,280 --> 01:16:39,350 software factory and you know automating 390 01:16:39,360 --> 01:16:41,510 everything that you can, whenever you 391 01:16:41,520 --> 01:16:43,350 correct your agent, try and do it by 392 01:16:43,360 --> 01:16:45,270 improving your codebase first. you know, 393 01:16:45,280 --> 01:16:47,030 start with a lint rule to prevent new 394 01:16:47,040 --> 01:16:49,430 occurrences of anti-atterns and then 395 01:16:49,440 --> 01:16:51,990 clean up those uh old patterns as well 396 01:16:52,000 --> 01:16:53,270 and then make sure that your 397 01:16:53,280 --> 01:16:56,790 verification story is really strong. 398 01:16:56,800 --> 01:16:58,870 A lot of this work is a prerequisite to 399 01:16:58,880 --> 01:17:01,430 automation anyway. And if your inner 400 01:17:01,440 --> 01:17:04,470 coding loop uh produces slop, then 401 01:17:04,480 --> 01:17:06,550 automating that loop just produces more 402 01:17:06,560 --> 01:17:08,550 slop and you need to throw more and more 403 01:17:08,560 --> 01:17:10,470 and more tokens at the problem, which we 404 01:17:10,480 --> 01:17:13,430 all know is very expensive. 405 01:17:13,440 --> 01:17:14,950 And once you've made these investments 406 01:17:14,960 --> 01:17:17,110 into, you know, code into your codebase, 407 01:17:17,120 --> 01:17:19,189 static analysis, runtime informal 408 01:17:19,199 --> 01:17:21,830 verification, you'll start to find that 409 01:17:21,840 --> 01:17:24,070 many things become possible with these 410 01:17:24,080 --> 01:17:26,310 foundations and that you can really 411 01:17:26,320 --> 01:17:28,630 easily build your own factory on top of 412 01:17:28,640 --> 01:17:31,030 these foundations. 413 01:17:31,040 --> 01:17:32,950 And when you're ready, Grabbot is a 414 01:17:32,960 --> 01:17:34,870 great way to build that uh outer loop. 415 01:17:34,880 --> 01:17:36,709 We have tons of connectors, routines 416 01:17:36,719 --> 01:17:38,950 that can kick off cursor cloud agents 417 01:17:38,960 --> 01:17:40,950 and projects uh which let you 418 01:17:40,960 --> 01:17:43,189 orchestrate thousands of hundreds or 419 01:17:43,199 --> 01:17:45,750 even thousands of agents. And as a quick 420 01:17:45,760 --> 01:17:47,669 example, you know, whenever we cut a new 421 01:17:47,679 --> 01:17:49,830 release now, Grockbot will automatically 422 01:17:49,840 --> 01:17:52,310 spin up a new cursor project for us that 423 01:17:52,320 --> 01:17:54,310 will fuzz the entire application, the 424 01:17:54,320 --> 01:17:56,790 entire build, and it will find bugs uh 425 01:17:56,800 --> 01:17:58,470 by running the app and using it like a 426 01:17:58,480 --> 01:18:01,270 real user. And then it's going to open 427 01:18:01,280 --> 01:18:03,350 fixed PRs for each of those bugs that it 428 01:18:03,360 --> 01:18:05,910 finds and it's going to automerge it as 429 01:18:05,920 --> 01:18:08,390 uh crazy as that sounds. And that's 430 01:18:08,400 --> 01:18:10,070 because our code base is good enough 431 01:18:10,080 --> 01:18:12,310 that pull requests tend to just be good 432 01:18:12,320 --> 01:18:14,550 by default. And most product code that 433 01:18:14,560 --> 01:18:16,790 we land no longer really requires deep 434 01:18:16,800 --> 01:18:20,070 human review. 435 01:18:20,080 --> 01:18:23,270 So when you have solid foundations and 436 01:18:23,280 --> 01:18:25,750 good organic architecture, you can turn 437 01:18:25,760 --> 01:18:28,390 slop into gold and you can ship with 438 01:18:28,400 --> 01:18:31,510 high quality uh high velocity without 439 01:18:31,520 --> 01:18:33,910 compromising on quality. My name is 440 01:18:33,920 --> 01:18:36,070 Potato, also known as Lauren. Thanks you 441 01:18:36,080 --> 01:18:38,000 again for listening to my talk.