# A New Age of Software Engineering

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[02:33:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9216s) Walden. I'm one of the co-founders and chief product officer at Cognition. We build an

[02:33:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9221s) AI product called Devon. I want to spend most of my time here today, though, kind of abstracting

[02:33:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9227s) a bit away from that and focusing on what are we doing internally as an engineering

[02:33:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9232s) team to move way faster.

[02:33:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9234s) And so the backdrop I would put for all this

[02:33:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9236s) is this is what our merged PR throughput looks like at Cognition.

[02:34:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9241s) The start of this is around 12 months ago.

[02:34:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9244s) So you're looking at about a snapshot of a year.

[02:34:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9246s) And over this last year, this blue chart here, this 40x,

[02:34:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9250s) is a 40x in the number of merged PRs

[02:34:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9253s) we are producing on our engineering team.

[02:34:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9255s) Now, you might also notice there's

[02:34:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9257s) a small black line along the bottom there.

[02:34:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9260s) It's easy to miss.

[02:34:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9260s) But that 2.7x, that is our engineering headcount growth in the same period.

[02:34:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9265s) So we have been hiring engineers, but each of our engineers is now more than 10x more productive than they were before.

[02:34:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9272s) And I would dare to say that these PRs are just as high quality, if not higher quality, than they were a year ago.

[02:34:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9278s) So really the crux of what I want to talk about today is what did we do to unlock this 40x engineering throughput?

[02:34:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9288s) And there's a lot of things we did from both the code base to the way we structure engineering

[02:34:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9293s) teams that I think are very practical steps you can take if you want to reproduce the

[02:34:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9297s) same results.

[02:34:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9299s) Many of you are probably familiar with the software factory.

[02:35:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9301s) How many of you guys have heard of software factory?

[02:35:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9302s) Let's see a raise of hands.

[02:35:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9304s) Cool.

[02:35:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9305s) So I'll start with the basics and then I'll work my way up to the more advanced stuff.

[02:35:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9309s) The basic idea of a software factory is, yeah, by default everything should be self-driving.

[02:35:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9315s) we have PRs that go from initial bug report to final merge and monitoring and maintenance

[02:35:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9321s) without a single human involved. Oftentimes human judgment is needed though, and that's where we

[02:35:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9325s) have escalations. But by default, our entire software development process is just run by

[02:35:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9330s) Devon from end to end. One of the things this means is that the role of the software engineer

[02:35:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9336s) has to change quite a bit. Around March of this year, we had this conversation with the engineering

[02:35:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9342s) team. And we said, look, if you are right now just picking off tickets off a linear board and then

[02:35:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9348s) sending that to an agent and implementing it, you're not going to really be set up for success

[02:35:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9351s) in the age of AI, because anything that is described and can be implemented will be able

[02:35:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9356s) to send to Devin. So really, everyone has to change the way that they're developing.

[02:36:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9362s) And the way we've enforced this is we've taken our larger teams and we've broken them up into

[02:36:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9366s) smaller pods. And the smaller pods not only are given swarms of agents to do work for them,

[02:36:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9372s) they are given more vertical responsibility.

[02:36:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9374s) So instead, you might have only two or three engineers

[02:36:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9378s) on a team at a time, but they're responsible for everything

[02:36:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9381s) beyond the engineering as well.

[02:36:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9382s) They're responsible for listening to customer bug reports.

[02:36:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9385s) They're responsible for doing QA testing.

[02:36:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9387s) They're responsible for doing the PMing and product management.

[02:36:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9390s) And basically, every engineer manages

[02:36:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9392s) their entire own linear board of things to do,

[02:36:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9395s) they send that to an agent to implement for them.

[02:36:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9397s) So it's really a fundamental change in the way

[02:36:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9399s) that our engineers do work.

[02:36:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9400s) This has been one of the things that we've implemented over the last year and we've been helping a lot of large organizations implement for themselves as well

[02:36:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9410s) Another thing I'd really like to highlight is we work from slack everywhere

[02:36:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9413s) so this is an example of what a customer bug report might look like they send me a video on slack and

[02:36:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9419s) We just forward bug reports straight into our internal channel and we tagged Devin and say hey, please fix this

[02:37:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9427s) Devon immediately comes up, shows up at PR, but then also sends back a video with the

[02:37:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9433s) fix and proof.

[02:37:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9436s) And this makes our lives so much easier because I literally don't have to leave Slack for

[02:37:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9439s) most of my work.

[02:37:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9440s) We've abstracted away all the development to be based on evidence, videos, and I can

[02:37:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9446s) just send the proof that the fix is ready right back to the customer without having to open

[02:37:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9450s) an IDE or even GitHub most of the time.

[02:37:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9454s) So this is kind of the end goal you want to reach which is a point where you can do your entire development lifecycle

[02:37:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9460s) Without opening up an IDE or without opening up a single coding agent

[02:37:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9466s) But to get to this point there's a lot of infrastructure

[02:37:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9470s) We had to change underneath and so this is the hard part that really every engineering team has to do if they want to get to that point

[02:37:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9476s) It's not just you purchase a coding agent

[02:37:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9477s) You got to go change the way you build so before when we test our applications

[02:38:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9481s) we'd be testing them against cloud deployments.

[02:38:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9483s) We'd have a staging environment, connect to a staging database,

[02:38:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9486s) connect to live servers, and then we'd

[02:38:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9488s) have our engineers manually go click through and verify

[02:38:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9490s) that the flows work end to end.

[02:38:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9492s) We did a big migration to move every service

[02:38:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9495s) to be able to run locally with Docker Compose.

[02:38:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9498s) We've made it so every third party service,

[02:38:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9500s) we have mocked versions to be able to run locally.

[02:38:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9502s) And these can all run inside a single virtual machine

[02:38:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9506s) that every Devon has access to.

[02:38:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9508s) And every coding agent in our company now

[02:38:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9510s) does not need to touch any live services at all to test their changes.

[02:38:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9515s) In fact, these VMs, we rebuild them every hour at the company.

[02:38:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9520s) Every hour, we pull the latest version of the code.

[02:38:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9522s) We rebuild the entire code base from scratch

[02:38:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9524s) to be able to run very quickly and easily from scratch.

[02:38:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9527s) And if these builds ever fail because your code base is rapidly changing,

[02:38:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9532s) we also have Devon be the first response there.

[02:38:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9534s) And they're always immediately fixing these VM snapshots.

[02:38:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9537s) So every time we spin up an agent from Slack,

[02:39:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9540s) it goes to a fresh machine on a fresh version of the code base with the latest updates.

[02:39:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9544s) And by the way, these agents, you need to teach them how to test your code base as well.

[02:39:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9550s) But the benefit of using cloud coding agents is that they can self-maintain their training loops.

[02:39:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9555s) So that last section there on testing, Devin will write its own skills for how to test our code base.

[02:39:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9562s) The code base will change over time, so the way you test it will change,

[02:39:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9564s) but Devin can keep on updating these skills so that it tests it over and over again.

[02:39:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9568s) So the end result of this is literally people do not need to open their laptops anymore to test any code changes

[02:39:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9573s) In fact several engineers at cognition have already moved to using iPads for the development process

[02:39:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9580s) By the way iPads are also more secure, you know a lot you see a lot of these hacks recently a lot of it is because of

[02:39:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9584s) DevOps and and people leaking keys who knows what's on your developer machine

[02:39:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9588s) You might have secrets all over the place when everything is in the cloud within these controlled VMs direction a much more secure spot

[02:39:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9596s) I've showed some basic versions of things you can test.

[02:39:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9598s) We've also tested advanced multiplayer

[02:40:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9600s) setups which is really cool. We've often times when we were developing Devon's ability to

[02:40:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9604s) join voice in Zoom calls, we actually had Devon, when it tests, create multiple accounts,

[02:40:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9610s) have them join the same Zoom call, all from within their one VM, and send voice memos back

[02:40:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9615s) and forth to make sure that the functionality is working. So you can do quite advanced setups

[02:40:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9619s) if you really push the limits of your code base.

[02:40:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9623s) And actually, on top of all this,

[02:40:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9625s) you obviously want recursive self-improvement.

[02:40:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9628s) I think people think about this from the model sense quite a lot.

[02:40:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9631s) But actually, if you think about it,

[02:40:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9634s) every single line of code that's being written,

[02:40:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9636s) every single PR that's being generated, every decision,

[02:40:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9639s) if you have a cloud agent setup,

[02:40:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9640s) that's all tracked across your entire code base.

[02:40:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9643s) So there's no reason why you shouldn't also be able to run agents

[02:40:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9645s) on top of that, do the data analysis,

[02:40:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9647s) and figure out where are we wasting time,

[02:40:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9649s) Where are we wasting tokens?

[02:40:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9651s) What things can be turned into scripts?

[02:40:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9653s) What tests can we be adding to improve this process?

[02:40:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9656s) And so we have a channel where Devon every day posts ideas about,

[02:41:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9660s) hey, if we change our node setup, we can shave a minute off of our CI.

[02:41:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9665s) If we change this lint script, we can catch 20% more of these bugs

[02:41:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9669s) before they ever hit AI code review.

[02:41:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9671s) And this has actually been a huge unlock

[02:41:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9673s) where you can just set up these systems,

[02:41:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9675s) and the agents will go and optimize your software factory for you.

[02:41:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9679s) This has been really exciting.

[02:41:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9681s) One of the things that I don't hear people talking about a lot,

[02:41:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9685s) but I think is really where things are going next,

[02:41:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9687s) is moving from the world where you're giving tasks to your agents

[02:41:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9691s) to giving jobs to your agents.

[02:41:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9693s) And this is a bit of a paradigm shift.

[02:41:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9695s) You really have to start treating these things like coworkers.

[02:41:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9698s) So what does giving a job to an agent look like?

[02:41:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9701s) Imagine I care about back-end performance.

[02:41:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9705s) Really, this is what the development flow of the future should look like.

[02:41:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9707s) You should send to your agent, hey, I want to improve P99 latency of my application.

[02:41:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9712s) Instead of you telling it what to do to do that, it should say, okay, P99 latency.

[02:41:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9717s) Let me add the telemetry to figure out where our P99 latency is today.

[02:42:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9721s) Let me set up alerts so I can figure out where we need to be in the future.

[02:42:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9724s) Let me go break down every single task we need to do to be able to move this latency.

[02:42:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9729s) Let me create data dog alerts.

[02:42:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9731s) Let me create weekly reports in Slack.

[02:42:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9733s) and let me send you updates on your phone every day on here's where P99 latency is and here's how much it's improved.

[02:42:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9739s) This is

[02:42:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9740s) fundamentally the type of systems that we encourage all of our engineers to be building at Cognition.

[02:42:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9745s) We don't even we don't even want you to be prompting your agents most of the time.

[02:42:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9748s) We want you to just be giving your agent goals that they can work toward over long durations of time.

[02:42:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9753s) And anything in your company that you think isn't measurable, find a way to measure it.

[02:42:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9758s) You know, do you care about your application performance?

[02:42:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9761s) go measure, you know, the latency of clicks on your application. If your

[02:42:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9764s) application has a lot of weird layout shifts, go measure the bounding boxes and

[02:42:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9768s) try to figure out how you can minimize layout shifts across your application.

[02:42:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9771s) There's obviously P99 latency, but there's also things like, you know,

[02:42:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9774s) onboarding retention. How much of what you guys do as engineers is things you

[02:42:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9779s) can boil down to a single number or metric, and I encourage you that if you

[02:43:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9782s) can do that, you should just set up an agent to go optimize that number 24-7.

[02:43:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9787s) So a great example of this is inference latency.

[02:43:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9790s) At Cognition, we're lucky to be the first receivers of NVIDIA's new class of Verorubin GPUs.

[02:43:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9798s) Kernels were not built for these GPUs when we received them.

[02:43:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9800s) So we had to, from scratch, figure out how do we build our own inference stack on top of this new hardware that's coming out.

[02:43:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9807s) And agents have allowed us to do this in a way that is much more efficient than before.

[02:43:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9812s) So we just tell it, hey, we care about the throughput, our TPM.

[02:43:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9816s) these GPUs go figure out how to push that to its limits and over the course of what is about I guess here like a week

[02:43:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9825s) The agent was able to go we allocated it a permanent set of GPUs to work on

[02:43:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9830s) It was able to go find different strategies test them the strategies that it found worked best

[02:43:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9835s) It was sent to a human engineer for final approval

[02:43:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9837s) But basically there's eight or nine different things that were roughly discovered

[02:44:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9843s) via just giving these agents these high-level goals,

[02:44:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9846s) and we were able to push the throughput

[02:44:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9849s) on these new and very Rubin GPUs

[02:44:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9851s) from what was worse than the new GB300s

[02:44:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9854s) to what is now significantly better

[02:44:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9856s) than the previous class of GB300 GPUs.

[02:44:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9859s) And again, this is all done

[02:44:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9860s) by just giving an end goal to your agents.

[02:44:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9866s) Another example that I like to share with folks,

[02:44:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9870s) we are a fairly small engineering team,

[02:44:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9873s) It means we do not have, oftentimes, the bandwidth to fully staff things that we might want to

[02:44:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9877s) do, like an Android application.

[02:44:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9880s) But the other day, one of our engineers on our team, Sam Yoke, he had this brilliant idea.

[02:44:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9884s) He said, okay, look, we have a team of people working on building a great iOS application.

[02:44:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9890s) Can't we just use that as a blueprint for our Android application and tell Devin, hey,

[02:44:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9895s) go maintain parity from our iOS application to our Android application end-to-end?

[02:45:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9901s) And that is exactly what he did.

[02:45:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9903s) But it wasn't just, hey, one dev in session to go do all this.

[02:45:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9907s) He said, set up permanent hooks against our code base to monitor the iOS application.

[02:45:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9912s) You know, go listen to the Slack channels for user feedback about our applications.

[02:45:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9918s) And then go recursively improve our application.

[02:45:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9920s) Go do live QA testing of every detail and making sure it all works.

[02:45:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9925s) Syncing these things live has been really cool.

[02:45:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9928s) And I think the final results here are actually quite sick.

[02:45:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9938s) So we really just gave our agent this one high-level prompt, which is, hey, go manage

[02:45:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9943s) the entire Android application.

[02:45:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9944s) And over the course of 44 days, we have a working application, which we'll be putting

[02:45:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9949s) out soon.

[02:45:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9950s) But as a time lapse, most days, there's not a single human message going to this agent

[02:45:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9955s) working on it.

[02:45:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9957s) And on those days, you know, it's merging upward of like 40, 50 PRs a day.

[02:46:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9963s) All this is being live QA tested against the iOS application, pixel for pixel, testing

[02:46:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9969s) what are, you know, escalating only the most important decisions to a human at the end

[02:46:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9973s) of the day.

[02:46:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9976s) And most of the time, our engineers don't even need to open Devon, they're just from

[02:46:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9979s) Slack, look at the recordings that it sends back to them.

[02:46:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9987s) So in the end, the human is barely doing anything in this loop.

[02:46:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9990s) This is how many messages a human actually had to send to our agent in this entire process.

[02:46:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9998s) Yeah.

[02:46:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=9998s) And it's not a slop application.

[02:46:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10000s) This is actually something that we will be going live with very soon.

[02:46:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10009s) So cool.

[02:46:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10010s) That's actually all I had.

[02:46:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10012s) I think the end message from all of this is if you set up the loops for your agents correctly,

[02:46:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10017s) you can do some pretty incredible stuff.

[02:47:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10020s) So I think really most teams are limited by the amount of ambition they have.

[02:47:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10024s) So it's time to be ambitious.

[02:47:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10026s) Thank you.