1 02:33:36,500 --> 02:33:41,160 Walden. I'm one of the co-founders and chief product officer at Cognition. We build an 2 02:33:41,160 --> 02:33:47,840 AI product called Devon. I want to spend most of my time here today, though, kind of abstracting 3 02:33:47,840 --> 02:33:52,320 a bit away from that and focusing on what are we doing internally as an engineering 4 02:33:52,320 --> 02:33:54,240 team to move way faster. 5 02:33:54,500 --> 02:33:56,520 And so the backdrop I would put for all this 6 02:33:56,520 --> 02:34:00,580 is this is what our merged PR throughput looks like at Cognition. 7 02:34:01,140 --> 02:34:04,580 The start of this is around 12 months ago. 8 02:34:04,640 --> 02:34:06,480 So you're looking at about a snapshot of a year. 9 02:34:06,660 --> 02:34:10,520 And over this last year, this blue chart here, this 40x, 10 02:34:10,700 --> 02:34:13,180 is a 40x in the number of merged PRs 11 02:34:13,180 --> 02:34:14,960 we are producing on our engineering team. 12 02:34:15,800 --> 02:34:17,720 Now, you might also notice there's 13 02:34:17,720 --> 02:34:20,300 a small black line along the bottom there. 14 02:34:20,380 --> 02:34:20,880 It's easy to miss. 15 02:34:20,880 --> 02:34:25,260 But that 2.7x, that is our engineering headcount growth in the same period. 16 02:34:25,440 --> 02:34:32,360 So we have been hiring engineers, but each of our engineers is now more than 10x more productive than they were before. 17 02:34:32,600 --> 02:34:38,340 And I would dare to say that these PRs are just as high quality, if not higher quality, than they were a year ago. 18 02:34:38,900 --> 02:34:46,320 So really the crux of what I want to talk about today is what did we do to unlock this 40x engineering throughput? 19 02:34:48,280 --> 02:34:53,060 And there's a lot of things we did from both the code base to the way we structure engineering 20 02:34:53,060 --> 02:34:57,460 teams that I think are very practical steps you can take if you want to reproduce the 21 02:34:57,460 --> 02:34:57,940 same results. 22 02:34:59,560 --> 02:35:01,300 Many of you are probably familiar with the software factory. 23 02:35:01,400 --> 02:35:02,720 How many of you guys have heard of software factory? 24 02:35:02,780 --> 02:35:03,540 Let's see a raise of hands. 25 02:35:04,500 --> 02:35:04,820 Cool. 26 02:35:05,040 --> 02:35:08,640 So I'll start with the basics and then I'll work my way up to the more advanced stuff. 27 02:35:09,400 --> 02:35:14,160 The basic idea of a software factory is, yeah, by default everything should be self-driving. 28 02:35:15,120 --> 02:35:21,240 we have PRs that go from initial bug report to final merge and monitoring and maintenance 29 02:35:21,240 --> 02:35:25,540 without a single human involved. Oftentimes human judgment is needed though, and that's where we 30 02:35:25,540 --> 02:35:30,240 have escalations. But by default, our entire software development process is just run by 31 02:35:30,240 --> 02:35:36,160 Devon from end to end. One of the things this means is that the role of the software engineer 32 02:35:36,160 --> 02:35:42,080 has to change quite a bit. Around March of this year, we had this conversation with the engineering 33 02:35:42,080 --> 02:35:48,240 team. And we said, look, if you are right now just picking off tickets off a linear board and then 34 02:35:48,240 --> 02:35:51,720 sending that to an agent and implementing it, you're not going to really be set up for success 35 02:35:51,720 --> 02:35:56,340 in the age of AI, because anything that is described and can be implemented will be able 36 02:35:56,340 --> 02:36:01,740 to send to Devin. So really, everyone has to change the way that they're developing. 37 02:36:02,240 --> 02:36:06,340 And the way we've enforced this is we've taken our larger teams and we've broken them up into 38 02:36:06,340 --> 02:36:11,460 smaller pods. And the smaller pods not only are given swarms of agents to do work for them, 39 02:36:12,080 --> 02:36:13,620 they are given more vertical responsibility. 40 02:36:14,640 --> 02:36:18,480 So instead, you might have only two or three engineers 41 02:36:18,480 --> 02:36:21,220 on a team at a time, but they're responsible for everything 42 02:36:21,220 --> 02:36:22,280 beyond the engineering as well. 43 02:36:22,400 --> 02:36:24,760 They're responsible for listening to customer bug reports. 44 02:36:25,120 --> 02:36:27,780 They're responsible for doing QA testing. 45 02:36:27,940 --> 02:36:30,500 They're responsible for doing the PMing and product management. 46 02:36:30,660 --> 02:36:32,480 And basically, every engineer manages 47 02:36:32,480 --> 02:36:34,560 their entire own linear board of things to do, 48 02:36:35,240 --> 02:36:37,240 they send that to an agent to implement for them. 49 02:36:37,340 --> 02:36:39,420 So it's really a fundamental change in the way 50 02:36:39,420 --> 02:36:40,680 that our engineers do work. 51 02:36:40,680 --> 02:36:47,780 This has been one of the things that we've implemented over the last year and we've been helping a lot of large organizations implement for themselves as well 52 02:36:50,100 --> 02:36:53,180 Another thing I'd really like to highlight is we work from slack everywhere 53 02:36:53,180 --> 02:36:58,020 so this is an example of what a customer bug report might look like they send me a video on slack and 54 02:36:59,560 --> 02:37:06,240 We just forward bug reports straight into our internal channel and we tagged Devin and say hey, please fix this 55 02:37:07,500 --> 02:37:13,320 Devon immediately comes up, shows up at PR, but then also sends back a video with the 56 02:37:13,320 --> 02:37:14,120 fix and proof. 57 02:37:16,460 --> 02:37:19,840 And this makes our lives so much easier because I literally don't have to leave Slack for 58 02:37:19,840 --> 02:37:20,420 most of my work. 59 02:37:20,560 --> 02:37:26,240 We've abstracted away all the development to be based on evidence, videos, and I can 60 02:37:26,240 --> 02:37:30,500 just send the proof that the fix is ready right back to the customer without having to open 61 02:37:30,500 --> 02:37:32,440 an IDE or even GitHub most of the time. 62 02:37:34,260 --> 02:37:40,200 So this is kind of the end goal you want to reach which is a point where you can do your entire development lifecycle 63 02:37:40,200 --> 02:37:43,920 Without opening up an IDE or without opening up a single coding agent 64 02:37:46,720 --> 02:37:50,280 But to get to this point there's a lot of infrastructure 65 02:37:50,280 --> 02:37:56,480 We had to change underneath and so this is the hard part that really every engineering team has to do if they want to get to that point 66 02:37:56,480 --> 02:37:57,840 It's not just you purchase a coding agent 67 02:37:57,840 --> 02:38:01,880 You got to go change the way you build so before when we test our applications 68 02:38:01,880 --> 02:38:03,720 we'd be testing them against cloud deployments. 69 02:38:03,940 --> 02:38:06,260 We'd have a staging environment, connect to a staging database, 70 02:38:06,720 --> 02:38:08,620 connect to live servers, and then we'd 71 02:38:08,620 --> 02:38:10,880 have our engineers manually go click through and verify 72 02:38:10,880 --> 02:38:12,180 that the flows work end to end. 73 02:38:12,560 --> 02:38:15,860 We did a big migration to move every service 74 02:38:15,860 --> 02:38:18,220 to be able to run locally with Docker Compose. 75 02:38:18,460 --> 02:38:20,280 We've made it so every third party service, 76 02:38:20,300 --> 02:38:22,460 we have mocked versions to be able to run locally. 77 02:38:22,940 --> 02:38:26,380 And these can all run inside a single virtual machine 78 02:38:26,960 --> 02:38:28,500 that every Devon has access to. 79 02:38:28,760 --> 02:38:30,740 And every coding agent in our company now 80 02:38:30,740 --> 02:38:33,960 does not need to touch any live services at all to test their changes. 81 02:38:35,940 --> 02:38:40,200 In fact, these VMs, we rebuild them every hour at the company. 82 02:38:40,300 --> 02:38:42,080 Every hour, we pull the latest version of the code. 83 02:38:42,240 --> 02:38:44,520 We rebuild the entire code base from scratch 84 02:38:44,520 --> 02:38:47,060 to be able to run very quickly and easily from scratch. 85 02:38:47,640 --> 02:38:51,620 And if these builds ever fail because your code base is rapidly changing, 86 02:38:52,680 --> 02:38:54,720 we also have Devon be the first response there. 87 02:38:54,780 --> 02:38:57,780 And they're always immediately fixing these VM snapshots. 88 02:38:57,960 --> 02:39:00,040 So every time we spin up an agent from Slack, 89 02:39:00,040 --> 02:39:03,840 it goes to a fresh machine on a fresh version of the code base with the latest updates. 90 02:39:04,980 --> 02:39:09,980 And by the way, these agents, you need to teach them how to test your code base as well. 91 02:39:10,380 --> 02:39:15,480 But the benefit of using cloud coding agents is that they can self-maintain their training loops. 92 02:39:15,700 --> 02:39:21,680 So that last section there on testing, Devin will write its own skills for how to test our code base. 93 02:39:22,140 --> 02:39:24,540 The code base will change over time, so the way you test it will change, 94 02:39:24,660 --> 02:39:28,040 but Devin can keep on updating these skills so that it tests it over and over again. 95 02:39:28,600 --> 02:39:33,780 So the end result of this is literally people do not need to open their laptops anymore to test any code changes 96 02:39:33,780 --> 02:39:39,080 In fact several engineers at cognition have already moved to using iPads for the development process 97 02:39:40,200 --> 02:39:44,560 By the way iPads are also more secure, you know a lot you see a lot of these hacks recently a lot of it is because of 98 02:39:44,560 --> 02:39:48,560 DevOps and and people leaking keys who knows what's on your developer machine 99 02:39:48,560 --> 02:39:53,820 You might have secrets all over the place when everything is in the cloud within these controlled VMs direction a much more secure spot 100 02:39:56,320 --> 02:39:58,280 I've showed some basic versions of things you can test. 101 02:39:58,380 --> 02:39:59,940 We've also tested advanced multiplayer 102 02:40:00,000 --> 02:40:04,700 setups which is really cool. We've often times when we were developing Devon's ability to 103 02:40:04,700 --> 02:40:09,760 join voice in Zoom calls, we actually had Devon, when it tests, create multiple accounts, 104 02:40:10,000 --> 02:40:15,960 have them join the same Zoom call, all from within their one VM, and send voice memos back 105 02:40:15,960 --> 02:40:19,900 and forth to make sure that the functionality is working. So you can do quite advanced setups 106 02:40:19,900 --> 02:40:21,700 if you really push the limits of your code base. 107 02:40:23,620 --> 02:40:25,720 And actually, on top of all this, 108 02:40:25,940 --> 02:40:27,940 you obviously want recursive self-improvement. 109 02:40:28,160 --> 02:40:30,820 I think people think about this from the model sense quite a lot. 110 02:40:31,020 --> 02:40:33,380 But actually, if you think about it, 111 02:40:34,080 --> 02:40:36,040 every single line of code that's being written, 112 02:40:36,320 --> 02:40:39,220 every single PR that's being generated, every decision, 113 02:40:39,500 --> 02:40:40,820 if you have a cloud agent setup, 114 02:40:40,940 --> 02:40:43,040 that's all tracked across your entire code base. 115 02:40:43,140 --> 02:40:45,420 So there's no reason why you shouldn't also be able to run agents 116 02:40:45,420 --> 02:40:47,480 on top of that, do the data analysis, 117 02:40:47,560 --> 02:40:49,620 and figure out where are we wasting time, 118 02:40:49,620 --> 02:40:50,800 Where are we wasting tokens? 119 02:40:51,180 --> 02:40:53,380 What things can be turned into scripts? 120 02:40:53,480 --> 02:40:55,900 What tests can we be adding to improve this process? 121 02:40:56,160 --> 02:41:00,020 And so we have a channel where Devon every day posts ideas about, 122 02:41:00,240 --> 02:41:04,760 hey, if we change our node setup, we can shave a minute off of our CI. 123 02:41:05,020 --> 02:41:09,580 If we change this lint script, we can catch 20% more of these bugs 124 02:41:09,580 --> 02:41:11,200 before they ever hit AI code review. 125 02:41:11,780 --> 02:41:13,540 And this has actually been a huge unlock 126 02:41:13,540 --> 02:41:15,180 where you can just set up these systems, 127 02:41:15,260 --> 02:41:18,480 and the agents will go and optimize your software factory for you. 128 02:41:19,140 --> 02:41:20,300 This has been really exciting. 129 02:41:21,740 --> 02:41:25,220 One of the things that I don't hear people talking about a lot, 130 02:41:25,280 --> 02:41:27,700 but I think is really where things are going next, 131 02:41:27,740 --> 02:41:31,220 is moving from the world where you're giving tasks to your agents 132 02:41:31,220 --> 02:41:32,960 to giving jobs to your agents. 133 02:41:33,960 --> 02:41:35,380 And this is a bit of a paradigm shift. 134 02:41:35,500 --> 02:41:37,660 You really have to start treating these things like coworkers. 135 02:41:38,040 --> 02:41:40,580 So what does giving a job to an agent look like? 136 02:41:41,720 --> 02:41:43,860 Imagine I care about back-end performance. 137 02:41:45,080 --> 02:41:47,860 Really, this is what the development flow of the future should look like. 138 02:41:47,860 --> 02:41:51,920 You should send to your agent, hey, I want to improve P99 latency of my application. 139 02:41:52,980 --> 02:41:57,160 Instead of you telling it what to do to do that, it should say, okay, P99 latency. 140 02:41:57,400 --> 02:42:00,720 Let me add the telemetry to figure out where our P99 latency is today. 141 02:42:01,260 --> 02:42:04,380 Let me set up alerts so I can figure out where we need to be in the future. 142 02:42:04,760 --> 02:42:09,480 Let me go break down every single task we need to do to be able to move this latency. 143 02:42:09,760 --> 02:42:11,480 Let me create data dog alerts. 144 02:42:11,560 --> 02:42:13,180 Let me create weekly reports in Slack. 145 02:42:13,180 --> 02:42:18,680 and let me send you updates on your phone every day on here's where P99 latency is and here's how much it's improved. 146 02:42:19,900 --> 02:42:20,660 This is 147 02:42:20,660 --> 02:42:24,940 fundamentally the type of systems that we encourage all of our engineers to be building at Cognition. 148 02:42:25,160 --> 02:42:28,160 We don't even we don't even want you to be prompting your agents most of the time. 149 02:42:28,280 --> 02:42:33,040 We want you to just be giving your agent goals that they can work toward over long durations of time. 150 02:42:33,560 --> 02:42:37,900 And anything in your company that you think isn't measurable, find a way to measure it. 151 02:42:38,360 --> 02:42:40,660 You know, do you care about your application performance? 152 02:42:41,080 --> 02:42:44,940 go measure, you know, the latency of clicks on your application. If your 153 02:42:44,940 --> 02:42:48,380 application has a lot of weird layout shifts, go measure the bounding boxes and 154 02:42:48,380 --> 02:42:51,000 try to figure out how you can minimize layout shifts across your application. 155 02:42:51,320 --> 02:42:54,500 There's obviously P99 latency, but there's also things like, you know, 156 02:42:54,560 --> 02:42:59,420 onboarding retention. How much of what you guys do as engineers is things you 157 02:42:59,420 --> 02:43:02,900 can boil down to a single number or metric, and I encourage you that if you 158 02:43:02,900 --> 02:43:06,580 can do that, you should just set up an agent to go optimize that number 24-7. 159 02:43:07,960 --> 02:43:10,480 So a great example of this is inference latency. 160 02:43:10,940 --> 02:43:16,640 At Cognition, we're lucky to be the first receivers of NVIDIA's new class of Verorubin GPUs. 161 02:43:18,280 --> 02:43:20,840 Kernels were not built for these GPUs when we received them. 162 02:43:20,880 --> 02:43:26,740 So we had to, from scratch, figure out how do we build our own inference stack on top of this new hardware that's coming out. 163 02:43:27,740 --> 02:43:31,440 And agents have allowed us to do this in a way that is much more efficient than before. 164 02:43:32,360 --> 02:43:35,920 So we just tell it, hey, we care about the throughput, our TPM. 165 02:43:36,580 --> 02:43:44,260 these GPUs go figure out how to push that to its limits and over the course of what is about I guess here like a week 166 02:43:45,440 --> 02:43:50,180 The agent was able to go we allocated it a permanent set of GPUs to work on 167 02:43:50,180 --> 02:43:55,160 It was able to go find different strategies test them the strategies that it found worked best 168 02:43:55,160 --> 02:43:57,980 It was sent to a human engineer for final approval 169 02:43:57,980 --> 02:44:03,480 But basically there's eight or nine different things that were roughly discovered 170 02:44:03,480 --> 02:44:06,380 via just giving these agents these high-level goals, 171 02:44:06,600 --> 02:44:09,460 and we were able to push the throughput 172 02:44:09,460 --> 02:44:11,280 on these new and very Rubin GPUs 173 02:44:11,280 --> 02:44:14,620 from what was worse than the new GB300s 174 02:44:14,620 --> 02:44:16,140 to what is now significantly better 175 02:44:16,140 --> 02:44:18,300 than the previous class of GB300 GPUs. 176 02:44:19,060 --> 02:44:20,380 And again, this is all done 177 02:44:20,380 --> 02:44:22,680 by just giving an end goal to your agents. 178 02:44:26,540 --> 02:44:29,800 Another example that I like to share with folks, 179 02:44:30,960 --> 02:44:33,040 we are a fairly small engineering team, 180 02:44:33,860 --> 02:44:37,880 It means we do not have, oftentimes, the bandwidth to fully staff things that we might want to 181 02:44:37,880 --> 02:44:39,280 do, like an Android application. 182 02:44:40,240 --> 02:44:43,980 But the other day, one of our engineers on our team, Sam Yoke, he had this brilliant idea. 183 02:44:44,380 --> 02:44:49,360 He said, okay, look, we have a team of people working on building a great iOS application. 184 02:44:50,080 --> 02:44:55,780 Can't we just use that as a blueprint for our Android application and tell Devin, hey, 185 02:44:55,880 --> 02:45:00,440 go maintain parity from our iOS application to our Android application end-to-end? 186 02:45:01,400 --> 02:45:03,380 And that is exactly what he did. 187 02:45:03,740 --> 02:45:07,340 But it wasn't just, hey, one dev in session to go do all this. 188 02:45:07,340 --> 02:45:12,360 He said, set up permanent hooks against our code base to monitor the iOS application. 189 02:45:12,840 --> 02:45:17,760 You know, go listen to the Slack channels for user feedback about our applications. 190 02:45:18,000 --> 02:45:20,700 And then go recursively improve our application. 191 02:45:20,920 --> 02:45:24,820 Go do live QA testing of every detail and making sure it all works. 192 02:45:25,700 --> 02:45:28,180 Syncing these things live has been really cool. 193 02:45:28,180 --> 02:45:31,940 And I think the final results here are actually quite sick. 194 02:45:38,720 --> 02:45:43,400 So we really just gave our agent this one high-level prompt, which is, hey, go manage 195 02:45:43,400 --> 02:45:44,620 the entire Android application. 196 02:45:44,940 --> 02:45:49,120 And over the course of 44 days, we have a working application, which we'll be putting 197 02:45:49,120 --> 02:45:49,520 out soon. 198 02:45:50,480 --> 02:45:55,840 But as a time lapse, most days, there's not a single human message going to this agent 199 02:45:55,840 --> 02:45:56,400 working on it. 200 02:45:57,440 --> 02:46:01,940 And on those days, you know, it's merging upward of like 40, 50 PRs a day. 201 02:46:03,440 --> 02:46:09,620 All this is being live QA tested against the iOS application, pixel for pixel, testing 202 02:46:09,620 --> 02:46:13,740 what are, you know, escalating only the most important decisions to a human at the end 203 02:46:13,740 --> 02:46:14,080 of the day. 204 02:46:16,340 --> 02:46:19,620 And most of the time, our engineers don't even need to open Devon, they're just from 205 02:46:19,620 --> 02:46:21,900 Slack, look at the recordings that it sends back to them. 206 02:46:27,960 --> 02:46:30,840 So in the end, the human is barely doing anything in this loop. 207 02:46:30,960 --> 02:46:36,220 This is how many messages a human actually had to send to our agent in this entire process. 208 02:46:38,140 --> 02:46:38,540 Yeah. 209 02:46:38,740 --> 02:46:40,900 And it's not a slop application. 210 02:46:40,900 --> 02:46:45,460 This is actually something that we will be going live with very soon. 211 02:46:49,540 --> 02:46:50,180 So cool. 212 02:46:50,480 --> 02:46:51,540 That's actually all I had. 213 02:46:52,040 --> 02:46:57,380 I think the end message from all of this is if you set up the loops for your agents correctly, 214 02:46:57,580 --> 02:46:59,860 you can do some pretty incredible stuff. 215 02:47:00,080 --> 02:47:04,300 So I think really most teams are limited by the amount of ambition they have. 216 02:47:04,360 --> 02:47:05,740 So it's time to be ambitious. 217 02:47:06,040 --> 02:47:06,420 Thank you.