Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
2:33:36Walden. I'm one of the co-founders and chief product officer at Cognition. We build an
2:33:41AI product called Devon. I want to spend most of my time here today, though, kind of abstracting
2:33:47a bit away from that and focusing on what are we doing internally as an engineering
2:33:52team to move way faster.
2:33:54And so the backdrop I would put for all this
2:33:56is this is what our merged PR throughput looks like at Cognition.
2:34:01The start of this is around 12 months ago.
2:34:04So you're looking at about a snapshot of a year.
2:34:06And over this last year, this blue chart here, this 40x,
2:34:10is a 40x in the number of merged PRs
2:34:13we are producing on our engineering team.
2:34:15Now, you might also notice there's
2:34:17a small black line along the bottom there.
2:34:20It's easy to miss.
2:34:20But that 2.7x, that is our engineering headcount growth in the same period.
2:34:25So we have been hiring engineers, but each of our engineers is now more than 10x more productive than they were before.
2:34:32And I would dare to say that these PRs are just as high quality, if not higher quality, than they were a year ago.
2:34:38So really the crux of what I want to talk about today is what did we do to unlock this 40x engineering throughput?
2:34:48And there's a lot of things we did from both the code base to the way we structure engineering
2:34:53teams that I think are very practical steps you can take if you want to reproduce the
2:34:57same results.
2:34:59Many of you are probably familiar with the software factory.
2:35:01How many of you guys have heard of software factory?
2:35:02Let's see a raise of hands.
2:35:04Cool.
2:35:05So I'll start with the basics and then I'll work my way up to the more advanced stuff.
2:35:09The basic idea of a software factory is, yeah, by default everything should be self-driving.
2:35:15we have PRs that go from initial bug report to final merge and monitoring and maintenance
2:35:21without a single human involved. Oftentimes human judgment is needed though, and that's where we
2:35:25have escalations. But by default, our entire software development process is just run by
2:35:30Devon from end to end. One of the things this means is that the role of the software engineer
2:35:36has to change quite a bit. Around March of this year, we had this conversation with the engineering
2:35:42team. And we said, look, if you are right now just picking off tickets off a linear board and then
2:35:48sending that to an agent and implementing it, you're not going to really be set up for success
2:35:51in the age of AI, because anything that is described and can be implemented will be able
2:35:56to send to Devin. So really, everyone has to change the way that they're developing.
2:36:02And the way we've enforced this is we've taken our larger teams and we've broken them up into
2:36:06smaller pods. And the smaller pods not only are given swarms of agents to do work for them,
2:36:12they are given more vertical responsibility.
2:36:14So instead, you might have only two or three engineers
2:36:18on a team at a time, but they're responsible for everything
2:36:21beyond the engineering as well.
2:36:22They're responsible for listening to customer bug reports.
2:36:25They're responsible for doing QA testing.
2:36:27They're responsible for doing the PMing and product management.
2:36:30And basically, every engineer manages
2:36:32their entire own linear board of things to do,
2:36:35they send that to an agent to implement for them.
2:36:37So it's really a fundamental change in the way
2:36:39that our engineers do work.
2:36:40This has been one of the things that we've implemented over the last year and we've been helping a lot of large organizations implement for themselves as well
2:36:50Another thing I'd really like to highlight is we work from slack everywhere
2:36:53so this is an example of what a customer bug report might look like they send me a video on slack and
2:36:59We just forward bug reports straight into our internal channel and we tagged Devin and say hey, please fix this
2:37:07Devon immediately comes up, shows up at PR, but then also sends back a video with the
2:37:13fix and proof.
2:37:16And this makes our lives so much easier because I literally don't have to leave Slack for
2:37:19most of my work.
2:37:20We've abstracted away all the development to be based on evidence, videos, and I can
2:37:26just send the proof that the fix is ready right back to the customer without having to open
2:37:30an IDE or even GitHub most of the time.
2:37:34So this is kind of the end goal you want to reach which is a point where you can do your entire development lifecycle
2:37:40Without opening up an IDE or without opening up a single coding agent
2:37:46But to get to this point there's a lot of infrastructure
2:37:50We had to change underneath and so this is the hard part that really every engineering team has to do if they want to get to that point
2:37:56It's not just you purchase a coding agent
2:37:57You got to go change the way you build so before when we test our applications
2:38:01we'd be testing them against cloud deployments.
2:38:03We'd have a staging environment, connect to a staging database,
2:38:06connect to live servers, and then we'd
2:38:08have our engineers manually go click through and verify
2:38:10that the flows work end to end.
2:38:12We did a big migration to move every service
2:38:15to be able to run locally with Docker Compose.
2:38:18We've made it so every third party service,
2:38:20we have mocked versions to be able to run locally.
2:38:22And these can all run inside a single virtual machine
2:38:26that every Devon has access to.
2:38:28And every coding agent in our company now
2:38:30does not need to touch any live services at all to test their changes.
2:38:35In fact, these VMs, we rebuild them every hour at the company.
2:38:40Every hour, we pull the latest version of the code.
2:38:42We rebuild the entire code base from scratch
2:38:44to be able to run very quickly and easily from scratch.
2:38:47And if these builds ever fail because your code base is rapidly changing,
2:38:52we also have Devon be the first response there.
2:38:54And they're always immediately fixing these VM snapshots.
2:38:57So every time we spin up an agent from Slack,
2:39:00it goes to a fresh machine on a fresh version of the code base with the latest updates.
2:39:04And by the way, these agents, you need to teach them how to test your code base as well.
2:39:10But the benefit of using cloud coding agents is that they can self-maintain their training loops.
2:39:15So that last section there on testing, Devin will write its own skills for how to test our code base.
2:39:22The code base will change over time, so the way you test it will change,
2:39:24but Devin can keep on updating these skills so that it tests it over and over again.
2:39:28So the end result of this is literally people do not need to open their laptops anymore to test any code changes
2:39:33In fact several engineers at cognition have already moved to using iPads for the development process
2:39:40By the way iPads are also more secure, you know a lot you see a lot of these hacks recently a lot of it is because of
2:39:44DevOps and and people leaking keys who knows what's on your developer machine
2:39:48You might have secrets all over the place when everything is in the cloud within these controlled VMs direction a much more secure spot
2:39:56I've showed some basic versions of things you can test.
2:39:58We've also tested advanced multiplayer
2:40:00setups which is really cool. We've often times when we were developing Devon's ability to
2:40:04join voice in Zoom calls, we actually had Devon, when it tests, create multiple accounts,
2:40:10have them join the same Zoom call, all from within their one VM, and send voice memos back
2:40:15and forth to make sure that the functionality is working. So you can do quite advanced setups
2:40:19if you really push the limits of your code base.
2:40:23And actually, on top of all this,
2:40:25you obviously want recursive self-improvement.
2:40:28I think people think about this from the model sense quite a lot.
2:40:31But actually, if you think about it,
2:40:34every single line of code that's being written,
2:40:36every single PR that's being generated, every decision,
2:40:39if you have a cloud agent setup,
2:40:40that's all tracked across your entire code base.
2:40:43So there's no reason why you shouldn't also be able to run agents
2:40:45on top of that, do the data analysis,
2:40:47and figure out where are we wasting time,
2:40:49Where are we wasting tokens?
2:40:51What things can be turned into scripts?
2:40:53What tests can we be adding to improve this process?
2:40:56And so we have a channel where Devon every day posts ideas about,
2:41:00hey, if we change our node setup, we can shave a minute off of our CI.
2:41:05If we change this lint script, we can catch 20% more of these bugs
2:41:09before they ever hit AI code review.
2:41:11And this has actually been a huge unlock
2:41:13where you can just set up these systems,
2:41:15and the agents will go and optimize your software factory for you.
2:41:19This has been really exciting.
2:41:21One of the things that I don't hear people talking about a lot,
2:41:25but I think is really where things are going next,
2:41:27is moving from the world where you're giving tasks to your agents
2:41:31to giving jobs to your agents.
2:41:33And this is a bit of a paradigm shift.
2:41:35You really have to start treating these things like coworkers.
2:41:38So what does giving a job to an agent look like?
2:41:41Imagine I care about back-end performance.
2:41:45Really, this is what the development flow of the future should look like.
2:41:47You should send to your agent, hey, I want to improve P99 latency of my application.
2:41:52Instead of you telling it what to do to do that, it should say, okay, P99 latency.
2:41:57Let me add the telemetry to figure out where our P99 latency is today.
2:42:01Let me set up alerts so I can figure out where we need to be in the future.
2:42:04Let me go break down every single task we need to do to be able to move this latency.
2:42:09Let me create data dog alerts.
2:42:11Let me create weekly reports in Slack.
2:42:13and let me send you updates on your phone every day on here's where P99 latency is and here's how much it's improved.
2:42:19This is
2:42:20fundamentally the type of systems that we encourage all of our engineers to be building at Cognition.
2:42:25We don't even we don't even want you to be prompting your agents most of the time.
2:42:28We want you to just be giving your agent goals that they can work toward over long durations of time.
2:42:33And anything in your company that you think isn't measurable, find a way to measure it.
2:42:38You know, do you care about your application performance?
2:42:41go measure, you know, the latency of clicks on your application. If your
2:42:44application has a lot of weird layout shifts, go measure the bounding boxes and
2:42:48try to figure out how you can minimize layout shifts across your application.
2:42:51There's obviously P99 latency, but there's also things like, you know,
2:42:54onboarding retention. How much of what you guys do as engineers is things you
2:42:59can boil down to a single number or metric, and I encourage you that if you
2:43:02can do that, you should just set up an agent to go optimize that number 24-7.
2:43:07So a great example of this is inference latency.
2:43:10At Cognition, we're lucky to be the first receivers of NVIDIA's new class of Verorubin GPUs.
2:43:18Kernels were not built for these GPUs when we received them.
2:43:20So we had to, from scratch, figure out how do we build our own inference stack on top of this new hardware that's coming out.
2:43:27And agents have allowed us to do this in a way that is much more efficient than before.
2:43:32So we just tell it, hey, we care about the throughput, our TPM.
2:43:36these GPUs go figure out how to push that to its limits and over the course of what is about I guess here like a week
2:43:45The agent was able to go we allocated it a permanent set of GPUs to work on
2:43:50It was able to go find different strategies test them the strategies that it found worked best
2:43:55It was sent to a human engineer for final approval
2:43:57But basically there's eight or nine different things that were roughly discovered
2:44:03via just giving these agents these high-level goals,
2:44:06and we were able to push the throughput
2:44:09on these new and very Rubin GPUs
2:44:11from what was worse than the new GB300s
2:44:14to what is now significantly better
2:44:16than the previous class of GB300 GPUs.
2:44:19And again, this is all done
2:44:20by just giving an end goal to your agents.
2:44:26Another example that I like to share with folks,
2:44:30we are a fairly small engineering team,
2:44:33It means we do not have, oftentimes, the bandwidth to fully staff things that we might want to
2:44:37do, like an Android application.
2:44:40But the other day, one of our engineers on our team, Sam Yoke, he had this brilliant idea.
2:44:44He said, okay, look, we have a team of people working on building a great iOS application.
2:44:50Can't we just use that as a blueprint for our Android application and tell Devin, hey,
2:44:55go maintain parity from our iOS application to our Android application end-to-end?
2:45:01And that is exactly what he did.
2:45:03But it wasn't just, hey, one dev in session to go do all this.
2:45:07He said, set up permanent hooks against our code base to monitor the iOS application.
2:45:12You know, go listen to the Slack channels for user feedback about our applications.
2:45:18And then go recursively improve our application.
2:45:20Go do live QA testing of every detail and making sure it all works.
2:45:25Syncing these things live has been really cool.
2:45:28And I think the final results here are actually quite sick.
2:45:38So we really just gave our agent this one high-level prompt, which is, hey, go manage
2:45:43the entire Android application.
2:45:44And over the course of 44 days, we have a working application, which we'll be putting
2:45:49out soon.
2:45:50But as a time lapse, most days, there's not a single human message going to this agent
2:45:55working on it.
2:45:57And on those days, you know, it's merging upward of like 40, 50 PRs a day.
2:46:03All this is being live QA tested against the iOS application, pixel for pixel, testing
2:46:09what are, you know, escalating only the most important decisions to a human at the end
2:46:13of the day.
2:46:16And most of the time, our engineers don't even need to open Devon, they're just from
2:46:19Slack, look at the recordings that it sends back to them.
2:46:27So in the end, the human is barely doing anything in this loop.
2:46:30This is how many messages a human actually had to send to our agent in this entire process.
2:46:38Yeah.
2:46:38And it's not a slop application.
2:46:40This is actually something that we will be going live with very soon.
2:46:49So cool.
2:46:50That's actually all I had.
2:46:52I think the end message from all of this is if you set up the loops for your agents correctly,
2:46:57you can do some pretty incredible stuff.
2:47:00So I think really most teams are limited by the amount of ambition they have.
2:47:04So it's time to be ambitious.
2:47:06Thank you.
