# Ship More as a Team of One with Kiro Crew

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[03:23:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12203s) Well, good afternoon, folks.

[03:23:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12204s) Wow, that's loud. You guys hear me okay? I take it yes.

[03:23:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12208s) I'm going to start with a question here today.

[03:23:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12210s) How many people feel like they're getting everything that they could possibly get from their agents today?

[03:23:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12216s) And that the promise of AI for your team has been delivered and you're satisfied with where you are?

[03:23:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12221s) Show of hands.

[03:23:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12223s) This is about what I usually see. Like, there's very few people.

[03:23:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12228s) And for us at Amazon, it's actually the very same.

[03:23:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12231s) Like when we first started this process, what happened is we were seeing between 10 and 20% gains,

[03:23:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12236s) as I show on the chart over here, you know, 1.1x to 1.2x.

[03:24:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12240s) And what we noticed was what was happening is teams were bolting AI onto a process that they already had.

[03:24:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12246s) So they're taking their normal process that they were doing,

[03:24:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12248s) they're trying to attach some AI here, attach some AI there,

[03:24:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12251s) and expecting that they're going to get this massive result out of it.

[03:24:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12254s) In addition, what we saw was teams were only shrinking the code generation phase.

[03:24:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12258s) And as you all know, you guys are all experts in this,

[03:24:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12261s) The code generation phase has gotten better and better and better as time has gone on.

[03:24:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12265s) But there's a lot more to software development than just code generation.

[03:24:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12269s) We did a study in June.

[03:24:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12271s) I published a blog post on it as well.

[03:24:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12275s) And in the study, we looked at teams who changed how they worked in our Amazon Stores division.

[03:24:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12280s) And we looked at what kind of returns were they getting on their AI investment.

[03:24:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12284s) Many of these teams were going slower before they were able to go faster.

[03:24:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12287s) But the median team was seeing a 4.5x return.

[03:24:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12291s) in terms of productivity improvement,

[03:24:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12293s) and specifically what this was,

[03:24:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12294s) was normalized development velocity

[03:24:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12296s) against that own team's historical baseline.

[03:24:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12299s) And we had some teams in the study

[03:25:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12300s) who exceeded 10x, right?

[03:25:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12302s) So we're talking significantly more productive.

[03:25:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12305s) And what they did was just,

[03:25:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12306s) they changed the way they were working.

[03:25:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12308s) But I see this still so often,

[03:25:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12310s) and just by your show of hands,

[03:25:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12311s) the same thing, right,

[03:25:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12312s) where people don't feel like that promise of AI

[03:25:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12315s) has quite been delivered yet.

[03:25:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12317s) Now, my name's Doug Clausen.

[03:25:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12319s) I'm a product manager on the Kiro team.

[03:25:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12321s) And I've been on this AI DevTool journey really since the beginning.

[03:25:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12324s) And I still see, like I said, teams who aren't quite getting what they expect.

[03:25:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12329s) Now, as a PM on Kiro, I get asked literally the same two questions all the time.

[03:25:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12334s) Like, this is a story of my life and meetings.

[03:25:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12336s) What are you shipping next?

[03:25:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12338s) And they want to know, like, what are the new features?

[03:25:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12340s) What new use cases might be enabled by those new features?

[03:25:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12343s) And, you know, I can walk through that, and that's great.

[03:25:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12345s) The other question, and oftentimes the more interesting one,

[03:25:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12348s) because everybody builds software slightly differently,

[03:25:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12350s) is are we doing this right?

[03:25:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12352s) People feel like they've hit some sort of local maximum

[03:25:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12355s) and they're scared that somebody else has figured out

[03:25:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12357s) how to use the AI better than they have

[03:25:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12359s) and they're just going to get surpassed.

[03:26:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12361s) We're actually going to spend today on the second question here,

[03:26:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12363s) not the first question.

[03:26:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12365s) I'm going to walk you through how one of our teams,

[03:26:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12367s) our Kuro Crew team, is shipping now thousands of PRs per week,

[03:26:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12371s) the big lessons we learned on the way

[03:26:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12373s) and sort of the process we went through to get to where we are today.

[03:26:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12376s) And then at the end of this, I'm going to leave you with a workshop

[03:26:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12379s) that if you want to try to build your own software factory in Kiro Crew,

[03:26:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12381s) you can go and do that as well.

[03:26:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12385s) Excuse me.

[03:26:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12386s) Now, first off, just some bit of context on Kiro.

[03:26:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12388s) So Kiro is an agentic development system

[03:26:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12390s) that lets you plan, build, and ship software with AI agents.

[03:26:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12394s) It works wherever you work.

[03:26:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12395s) So up here I'm showing you we have an IDE client,

[03:26:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12398s) a CLI, a web, and a mobile client.

[03:26:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12400s) And then I just talked just briefly, we have Kiro Crew as well.

[03:26:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12404s) We support a variety of models.

[03:26:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12406s) So we have both the Anthropic models and the OpenAI models in Kiro, as well as some openweight models.

[03:26:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12411s) I'll talk a little bit later how we use that and how that's been beneficial to us.

[03:26:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12416s) And just like the rest of AWS, we really push the barrier on cost performance.

[03:27:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12420s) So when we start saving money and we make the models cheaper to run and the agents cheaper to run, we end up passing those savings on to you.

[03:27:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12427s) So we're really happy with where Kiro has gotten on that Pareto front of sort of cost performance and cost effectiveness.

[03:27:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12433s) Excuse me.

[03:27:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12433s) Now, like I said, we're going to spend this day talking about KiroCrew, and KiroCrew

[03:27:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12437s) was built to run many Kiro agents at the same time.

[03:27:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12440s) It's all open source.

[03:27:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12441s) It actually started internally as an internal Amazon product where some engineers grabbed

[03:27:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12447s) our CLI and they're like, hey, we want to extend this.

[03:27:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12449s) It was during the whole OpenClaw craze, if anybody has their Mac minis at home, they've

[03:27:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12454s) been running those.

[03:27:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12454s) It happened during that craze where people wanted to figure out how they could get these

[03:27:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12458s) agents to be more like personal assistants.

[03:27:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12459s) And so this MeshClaw product internally was developed,

[03:27:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12462s) got around 40,000 new internal Amazon developers on it

[03:27:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12466s) in the first two months.

[03:27:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12468s) And so us at the Kiro team were like,

[03:27:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12469s) hey, since it was built on Kiro,

[03:27:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12471s) we should take that and extend it and release it.

[03:27:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12473s) And so we've done that now.

[03:27:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12474s) And Kiro Crew is released.

[03:27:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12476s) It's open source.

[03:27:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12478s) And if I talk about what have we seen with the Kiro Crew team

[03:28:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12482s) and our team in terms of building and developing this,

[03:28:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12485s) we shipped on August 4th externally.

[03:28:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12487s) And again, it's open source,

[03:28:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12489s) so you can go look at all the source and look at all the PRs that came through.

[03:28:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12492s) We go by one philosophy, that if we don't use it and love it, then we don't ship it.

[03:28:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12497s) Since we launched, we've merged about 16,500 PRs with a very small team.

[03:28:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12502s) Just last week, there was over 3,000 in a single week alone.

[03:28:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12506s) Take contributions from external entities, both internal to Amazon and external to Amazon,

[03:28:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12511s) people who just want to contribute and take a good idea with open source.

[03:28:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12515s) And if you look at the chart here, we sort of leveled out at around, you know, if you

[03:28:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12519s) look at just our shipping speed, we sort of leveled out at around high 100, 600 to 800,

[03:28:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12525s) sort of the first few months after we shipped.

[03:28:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12528s) And then you'll see in like 928, right, all of a sudden we break 1,000 again, and then

[03:28:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12532s) we're sort of hovering around, I'll average it out in my head and say 1.2K-ish, right?

[03:28:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12537s) And then you can see we made another step function improvement.

[03:29:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12540s) And while I'm showing you PRs on this chart, what we really care about among the team is

[03:29:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12544s) not necessarily the PRs, but we care about how many sessions can an individual developer

[03:29:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12550s) operate at the same time.

[03:29:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12552s) And I want to be clear that this is not a one size fits all thing, so as I walk through

[03:29:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12556s) this, just keep that in mind that what works for us may not work for you, and so you should

[03:29:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12560s) challenge what are you doing that you could learn from this.

[03:29:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12565s) Last thing on this one, about one in ten of these is features.

[03:29:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12568s) So I always get asked, like, man, 3,300 features,

[03:29:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12571s) that's a lot of features in a week.

[03:29:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12573s) Well, no, it's not all features, right?

[03:29:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12574s) This is what shipping with speed looks like.

[03:29:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12576s) A lot of, so I get asked, okay,

[03:29:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12578s) so about one in 10 of these is a feature, so about 10%.

[03:29:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12582s) And so if 10% are a feature, what are the other 90%?

[03:29:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12586s) Well, for us, it's tests.

[03:29:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12587s) Tests is a big part of what we do.

[03:29:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12590s) It's, I'll say, paper cut fixes for things we've already shipped

[03:29:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12593s) that we're just sort of learning about how it's getting used

[03:29:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12596s) and we're massaging it a little bit.

[03:29:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12598s) It's bug fixes as well.

[03:29:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12599s) And that's

[03:30:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12600s) really what shipping at a high speed with velocity looks like, right?

[03:30:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12603s) It's not all new features, but if you just think about 10% of our features, there's still

[03:30:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12606s) hundreds of features, new features being shipped over this timeline.

[03:30:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12611s) If I think about sort of how did we evolve with our development cycle, we started, if

[03:30:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12617s) I go back to when our CLI product initially launched, it was one session, one person.

[03:30:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12621s) So I'm sure many of us experienced this

[03:30:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12623s) when these new agents came out,

[03:30:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12625s) and I was typing, and it would then work,

[03:30:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12627s) and then it would stop working,

[03:30:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12628s) and then I would type again, right?

[03:30:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12629s) We'd just go back and forth.

[03:30:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12631s) And then from that, so that was one session, right?

[03:30:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12633s) I said we're trying to optimize sessions per person.

[03:30:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12636s) Then we moved to parallel sessions,

[03:30:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12637s) where instead of just running one session,

[03:30:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12639s) I'm looking at, you know, maybe I'm running two or three,

[03:30:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12641s) but I'm running them in the same manner, right?

[03:30:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12643s) And the problem with both of these is that

[03:30:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12645s) if you make yourself the feedback loop,

[03:30:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12648s) you are now the bottleneck, right?

[03:30:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12649s) These agents got faster long before you're going to be able to keep up.

[03:30:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12653s) And so these first two sessions, they're just going to kill in one chart here,

[03:30:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12656s) where it's, you know, we got to two or three sessions per person,

[03:30:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12658s) but we sort of topped out.

[03:31:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12660s) Now, stage three introduced memory and schedules.

[03:31:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12663s) And with a memory, a new session now all of a sudden knows your conventions, right?

[03:31:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12668s) So implicit memories.

[03:31:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12669s) And then schedules make it so you don't have to kick all of the work off on your own.

[03:31:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12673s) Things can happen on a schedule.

[03:31:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12675s) The thing we learned here was that schedules only help if your machine is on.

[03:31:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12679s) So in the previous phases, most of our engineers were running locally, right, as I'm sure many of you do still today.

[03:31:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12684s) But moving to remote ends up making it so you no longer have to worry about, is it going to run?

[03:31:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12688s) Because I have an always-on machine, something we're very good at at AWS.

[03:31:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12692s) So obviously we are running remote.

[03:31:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12695s) But what happened then is this allowed us to sort of, I'll say, build up to between 10 and 20 sessions, right?

[03:31:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12700s) And then we sort of plateaued again.

[03:31:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12702s) We were unable to sort of break that barrier with just memory and schedules.

[03:31:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12706s) because much of what we were doing

[03:31:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12708s) was still being the glue between the sessions, right?

[03:31:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12712s) And then if we, our next phase was a pipeline,

[03:31:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12715s) but not a CICD pipeline like you might be thinking,

[03:31:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12717s) but instead more like an assembly line.

[03:32:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12720s) I actually have, my previous life, I worked in aerospace,

[03:32:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12723s) so a lot of assembly lines,

[03:32:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12724s) and the hard thing with assembly lines is nonstandard work, right?

[03:32:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12727s) So if everything is exactly the same as it flows through the system,

[03:32:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12730s) an assembly line works great.

[03:32:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12731s) As soon as you get nonstandard work,

[03:32:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12733s) all of a sudden it becomes difficult,

[03:32:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12734s) and I'm sure all of you know this

[03:32:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12736s) but software doesn't follow the exact same

[03:32:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12738s) path every time

[03:32:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12739s) and so what we found our own teams doing then

[03:32:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12742s) was this, sorry, this got us to maybe like

[03:32:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12744s) 30, 20, 30, 40

[03:32:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12746s) sessions-ish but what happened

[03:32:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12748s) was is our team was then trying to

[03:32:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12750s) unblock the assembly line or the pipeline, right

[03:32:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12752s) so as things came up if we

[03:32:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12754s) thought something was a small issue and then it turned into

[03:32:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12756s) a big issue, sometimes the assembly line would be

[03:32:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12758s) stuck, somebody would have to come in and unblock

[03:32:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12760s) that thing, maybe you restart it, maybe you replan

[03:32:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12762s) it. And so this is only going to carry us so far because, again, we are in the midst

[03:32:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12767s) of the agent. We're blocking the agent more than the agent wants to work.

[03:32:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12773s) And so what did we do for that? Well, we launched a thing called Ultra Mode. Ultra Mode

[03:32:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12778s) is still, just to be clear, if you go try it and you look for it, it's still in the developer

[03:33:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12783s) preview, so it's behind a feature flag. But because it's open source, the code is all

[03:33:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12786s) there. If you want to go try it, you can try it out. As I said up front, if we don't love

[03:33:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12790s) it, we don't ship it, so we're still sort of finishing, putting the finishing touches

[03:33:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12793s) on this one.

[03:33:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12795s) But I wanted to talk about it anyway, because it takes you from being, what we did was,

[03:33:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12799s) if you look at this middle part here, where we are the person driving the effort, these

[03:33:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12804s) are the things that we might have been doing.

[03:33:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12806s) So watching for problems was one of the biggest ones, and then trying to fix the assembly

[03:33:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12809s) line when the problem came up.

[03:33:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12811s) Maybe you're judging a result, or you're deciding what to do next.

[03:33:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12815s) What Ultra Mode does is it does that job for you, right?

[03:33:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12818s) So you put an agent on top of your agents that then is responsible for doing that, defining

[03:33:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12822s) the goal based on whatever you're trying to accomplish, starting things, watching for

[03:33:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12826s) problems, unsticking things.

[03:33:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12828s) And this is getting us today to 50 plus sessions.

[03:33:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12832s) And so if you think about where we started with a single session and a single person,

[03:33:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12836s) now if a single developer is doing 50 plus sessions, that's how you get that sort of

[03:34:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12840s) the volume in the PR chart I showed you earlier.

[03:34:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12844s) And we can already see the next wall that's coming.

[03:34:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12847s) The orchestrators working with other orchestrators,

[03:34:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12850s) and then the governance around what you want that to be

[03:34:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12852s) and the guardrails become even more important, right?

[03:34:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12856s) So again, that's how we got the volume.

[03:34:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12858s) Those were the five phases we went through.

[03:34:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12860s) Ultra mode has been a game changer for us,

[03:34:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12862s) as was the assembly line.

[03:34:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12863s) And I invite you to try those ideas

[03:34:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12865s) if you think they might work for you.

[03:34:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12867s) The next thing I wanted to talk about, though,

[03:34:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12869s) was what did we learn from this?

[03:34:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12871s) Like, what are the biggest things that prevented us from shipping quickly?

[03:34:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12875s) The first one is test everything.

[03:34:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12878s) And now this is no, this is not a new lesson, right?

[03:34:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12880s) Everybody in software knows this.

[03:34:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12881s) We should be doing this.

[03:34:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12883s) But I still see people too often discount the value this is providing.

[03:34:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12888s) And this is gonna build in with the next one I have behind this around reviews.

[03:34:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12892s) Like, if you think about where do you get confidence to ship something?

[03:34:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12894s) I think many people still feel like, hey, I get confidence because I looked at all the

[03:34:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12898s) code and I know exactly what that code does. Again, if you do that, like maybe that's a

[03:35:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12903s) good thing, but where we have gotten our confidence, and Amazon's always been very test-driven,

[03:35:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12908s) is from the tests. Because you might not be able to look at something every time, but

[03:35:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12912s) you can certainly test something every time in the exact same way and test the result,

[03:35:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12917s) right? And come out with the result. And so for us, again, quality really becomes a testing

[03:35:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12922s) problem and we're spending a significant portion of our time, around 20% just on testing today.

[03:35:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12929s) The other big thing we learned, and this is a bigger one, and also a contentious topic,

[03:35:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12933s) is, is reviews.

[03:35:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12935s) So quick show of hands, how many people have done a review with an agent, thinking that

[03:35:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12939s) there's going to be like one or two comments, and then you come back and you're like, oh

[03:35:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12943s) my god, like this is terrible code, right?

[03:35:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12947s) There's comments everywhere.

[03:35:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12948s) We had the same problem.

[03:35:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12949s) So we originally wrote a review agent that would review everything, and our goal was

[03:35:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12953s) to fix every single comment.

[03:35:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12954s) Well, as you guys, I heard the laughter, so as you expect, you submit changes, so you

[03:36:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12960s) submit a PR, you get the feedback on that PR, you come back, you submit it again after you

[03:36:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12965s) fix it, you come back and now it's three new things that weren't even tested, weren't

[03:36:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12968s) even told about before and it's four things on the things you already fixed.

[03:36:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12971s) You submit it again and this cycle just continues to repeat.

[03:36:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12974s) So what we've done with reviews is now we categorize the reviews or triage them into

[03:36:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12979s) blocking and non-blocking issues.

[03:36:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12981s) So, for example, something that breaks the build,

[03:36:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12985s) if there's something obviously that breaks the build,

[03:36:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12986s) that's a blocking thing,

[03:36:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12987s) and we need to go fix that before we merge this PR.

[03:36:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12990s) Changing a variable name, not a big deal.

[03:36:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12992s) We'll take care of that later,

[03:36:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12993s) so we let those things slide through.

[03:36:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12996s) The other thing that we've done with reviews

[03:36:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=12998s) is we want to have fast feedback for an agent,

[03:36:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13000s) and as you can see by the process here

[03:36:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13002s) that I'll walk you through in a second,

[03:36:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13003s) this is not that fast.

[03:36:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13004s) And so we have a fast gate

[03:36:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13005s) that really double-checks things really quickly

[03:36:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13007s) to make sure that things are in the right folders

[03:36:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13009s) before we move on to our reviews.

[03:36:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13011s) We have six reviewers, actually, that run.

[03:36:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13013s) Two code reviewers, both in different models,

[03:36:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13016s) with different model families.

[03:36:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13018s) And the intent there is to really find,

[03:37:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13020s) to make sure you don't have a model gap, right,

[03:37:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13022s) so that the model wasn't trained on something that we don't catch.

[03:37:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13024s) This is where we use the Anthropic and OpenAI models

[03:37:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13027s) or an open weight model to be those levers.

[03:37:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13030s) We run a design review that asks,

[03:37:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13033s) does this thing have the right shape?

[03:37:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13034s) Meaning, is what I'm building,

[03:37:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13036s) like if I'm adding another cache in this PR,

[03:37:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13038s) can the cache I already have take care of this issue or not.

[03:37:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13042s) We have a first principles review that says,

[03:37:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13044s) is this change addressing the real problem?

[03:37:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13047s) We have a UX review as well,

[03:37:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13049s) that if you're making it, let's say you're creating a new error message,

[03:37:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13052s) we have an agent decide whether or not that error message,

[03:37:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13054s) the human will know what to do with that error message

[03:37:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13056s) based on what we've published.

[03:37:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13058s) And then we're consistently tightening our security posture.

[03:37:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13061s) Now, each one of these reviewers gets one vote.

[03:37:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13063s) It's either yes or no. It's red or it's green.

[03:37:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13065s) If everything's green, it goes on to merge.

[03:37:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13067s) If not, we go to a prepare PR skill that's in the repo,

[03:37:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13070s) so you can check the skill out if you want.

[03:37:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13071s) And that prepare PR skill then tries to resolve the issues as we go.

[03:37:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13077s) The third lesson, and sorry, on the reviews, one more thing.

[03:38:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13080s) Like I said, we've stopped reviewing code a long time ago.

[03:38:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13083s) We have moved on, and what we're responsible for is not the code itself.

[03:38:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13086s) See, this actually goes back. It does.

[03:38:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13088s) What we're responsible for is the review record.

[03:38:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13090s) Did it follow the process as we expected?

[03:38:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13092s) And did we get the results we wanted?

[03:38:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13094s) If not, we fixed the review process.

[03:38:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13097s) Two other things. So, one, the over-engineering problem. So we have a specific role to teach

[03:38:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13104s) our agents to subtract, not just add. And so we try to remove code from the process

[03:38:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13108s) as we go to prevent that over-engineering. And we use that same philosophy on our roadmap

[03:38:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13112s) as well, where saying yes is becoming very cheap or has become very cheap, right? It's

[03:38:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13116s) very easy for us now with code to say, like, yes, I'll do that and I'll do all of these

[03:38:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13120s) things. Whereas we used to have to be very particular about what we were going to build.

[03:38:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13124s) The problem with saying yes all the time

[03:38:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13126s) is that you might lose your product shape,

[03:38:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13128s) and we're already seeing this in KuroKribit,

[03:38:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13130s) that it's moving so fast and so quickly

[03:38:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13132s) that you don't want your users to get caught in the noise.

[03:38:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13135s) And so, again, use the same principle on the roadmap

[03:38:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13137s) as what we do.

[03:38:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13139s) And then last thing on lessons is

[03:39:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13141s) you're not done once you merge a PR, right?

[03:39:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13144s) There's still other phases,

[03:39:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13145s) and so for us, merging is not the last check that we do.

[03:39:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13148s) Instead, we run a release gate

[03:39:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13149s) where the agent will download the repo,

[03:39:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13152s) It installs it, runs it, and then checks the core flows

[03:39:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13154s) to make sure that the overall system is still in good shape

[03:39:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13157s) before we would do a release.

[03:39:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13158s) And then we do security all the time as well.

[03:39:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13161s) So we have every hour, every day, an always-on,

[03:39:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13163s) every hour against our security guidance,

[03:39:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13165s) every day against an AppSec review,

[03:39:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13167s) and then an always-on pen test agent.

[03:39:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13170s) So again, merging is not the end of the process.

[03:39:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13172s) You have to crunch everything down.

[03:39:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13175s) So I'm going to come back to where I started.

[03:39:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13177s) Are you getting everything from your agent?

[03:39:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13179s) and many of us don't feel like we are yet.

[03:39:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13182s) And what I'm going to offer here today,

[03:39:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13183s) or what I've tried to offer is,

[03:39:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13185s) the agent was fast enough to work on this a long time ago

[03:39:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13189s) and it's only getting better and better and better.

[03:39:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13191s) And so what we have been doing is challenging how we work

[03:39:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13194s) and figuring out where are we preventing the agent

[03:39:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13196s) from working as quickly as possible.

[03:39:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13198s) And I've given you numerous

[03:40:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13200s) examples today.

[03:40:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13201s) Again, I'll just call it a couple of them here.

[03:40:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13203s) So it started with our attention span, right?

[03:40:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13205s) When I was the feedback loop, I could no longer ship because I was the feedback loop.

[03:40:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13209s) When I wasn't there, the agent stopped working.

[03:40:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13211s) The laptop lid closing.

[03:40:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13212s) All of a sudden, you lose your connection.

[03:40:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13214s) Moving to remote can solve that, right?

[03:40:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13216s) The fixed pipeline with a fixed shape that can't respond when things go slightly off path.

[03:40:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13221s) Again, how do you fix that?

[03:40:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13222s) We fixed it with an orchestrator.

[03:40:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13224s) ultra-mode. The endless

[03:40:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13226s) reviews that I think I've gone into probably

[03:40:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13228s) more depth than you were interested in, but

[03:40:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13229s) the endless review cycle, and then the loop that only

[03:40:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13232s) adds, right? The loop that only adds things

[03:40:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13234s) to it and doesn't try to subtract.

[03:40:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13238s) And so

[03:40:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13238s) I'll leave you with, I also want to be

[03:40:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13240s) clear that this is going to be a

[03:40:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13243s) continual adjustment,

[03:40:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13244s) right? This is not, like, this may be what works for us

[03:40:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13246s) today. I promise you another month we're going to find

[03:40:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13248s) something else that works, and we're going to continue to evolve

[03:40:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13250s) significantly. But like I said in the

[03:40:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13252s) beginning, the pace that we're finally able to deliver at, I feel like for us, is that finally

[03:40:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13257s) we're meeting that promise of AI, and I still see the disparity among other teams.

[03:41:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13263s) And so, like I said, I'm going to leave you with this. If you're interested in trying Kiro Crew and

[03:41:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13268s) building your own software factory, scan this QR code. It's a workshop. We have a pretty generous

[03:41:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13273s) free trial on Kiro, so you can go sign up for a free trial, go install Kiro Crew, and you'll be

[03:41:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13279s) off and running.

[03:41:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13281s) And if you don't, if you want to try, like I said, if you want to try CreoCure, you can

[03:41:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13283s) do that.

[03:41:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13284s) The same free trial works across all of the products.

[03:41:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13286s) So the IDE, the CLI, the web client, the mobile client, try them all with that free trial.

[03:41:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13291s) And with that, I'm excited to see what you all build.

[03:41:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13294s) I appreciate the time.

[03:41:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13294s) Thank you.

[03:41:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13295s) Enjoy the rest of the conference.