Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
3:23:23Well, good afternoon, folks.
3:23:24Wow, that's loud. You guys hear me okay? I take it yes.
3:23:28I'm going to start with a question here today.
3:23:30How many people feel like they're getting everything that they could possibly get from their agents today?
3:23:36And that the promise of AI for your team has been delivered and you're satisfied with where you are?
3:23:41Show of hands.
3:23:43This is about what I usually see. Like, there's very few people.
3:23:48And for us at Amazon, it's actually the very same.
3:23:51Like when we first started this process, what happened is we were seeing between 10 and 20% gains,
3:23:56as I show on the chart over here, you know, 1.1x to 1.2x.
3:24:00And what we noticed was what was happening is teams were bolting AI onto a process that they already had.
3:24:06So they're taking their normal process that they were doing,
3:24:08they're trying to attach some AI here, attach some AI there,
3:24:11and expecting that they're going to get this massive result out of it.
3:24:14In addition, what we saw was teams were only shrinking the code generation phase.
3:24:18And as you all know, you guys are all experts in this,
3:24:21The code generation phase has gotten better and better and better as time has gone on.
3:24:25But there's a lot more to software development than just code generation.
3:24:29We did a study in June.
3:24:31I published a blog post on it as well.
3:24:35And in the study, we looked at teams who changed how they worked in our Amazon Stores division.
3:24:40And we looked at what kind of returns were they getting on their AI investment.
3:24:44Many of these teams were going slower before they were able to go faster.
3:24:47But the median team was seeing a 4.5x return.
3:24:51in terms of productivity improvement,
3:24:53and specifically what this was,
3:24:54was normalized development velocity
3:24:56against that own team's historical baseline.
3:24:59And we had some teams in the study
3:25:00who exceeded 10x, right?
3:25:02So we're talking significantly more productive.
3:25:05And what they did was just,
3:25:06they changed the way they were working.
3:25:08But I see this still so often,
3:25:10and just by your show of hands,
3:25:11the same thing, right,
3:25:12where people don't feel like that promise of AI
3:25:15has quite been delivered yet.
3:25:17Now, my name's Doug Clausen.
3:25:19I'm a product manager on the Kiro team.
3:25:21And I've been on this AI DevTool journey really since the beginning.
3:25:24And I still see, like I said, teams who aren't quite getting what they expect.
3:25:29Now, as a PM on Kiro, I get asked literally the same two questions all the time.
3:25:34Like, this is a story of my life and meetings.
3:25:36What are you shipping next?
3:25:38And they want to know, like, what are the new features?
3:25:40What new use cases might be enabled by those new features?
3:25:43And, you know, I can walk through that, and that's great.
3:25:45The other question, and oftentimes the more interesting one,
3:25:48because everybody builds software slightly differently,
3:25:50is are we doing this right?
3:25:52People feel like they've hit some sort of local maximum
3:25:55and they're scared that somebody else has figured out
3:25:57how to use the AI better than they have
3:25:59and they're just going to get surpassed.
3:26:01We're actually going to spend today on the second question here,
3:26:03not the first question.
3:26:05I'm going to walk you through how one of our teams,
3:26:07our Kuro Crew team, is shipping now thousands of PRs per week,
3:26:11the big lessons we learned on the way
3:26:13and sort of the process we went through to get to where we are today.
3:26:16And then at the end of this, I'm going to leave you with a workshop
3:26:19that if you want to try to build your own software factory in Kiro Crew,
3:26:21you can go and do that as well.
3:26:25Excuse me.
3:26:26Now, first off, just some bit of context on Kiro.
3:26:28So Kiro is an agentic development system
3:26:30that lets you plan, build, and ship software with AI agents.
3:26:34It works wherever you work.
3:26:35So up here I'm showing you we have an IDE client,
3:26:38a CLI, a web, and a mobile client.
3:26:40And then I just talked just briefly, we have Kiro Crew as well.
3:26:44We support a variety of models.
3:26:46So we have both the Anthropic models and the OpenAI models in Kiro, as well as some openweight models.
3:26:51I'll talk a little bit later how we use that and how that's been beneficial to us.
3:26:56And just like the rest of AWS, we really push the barrier on cost performance.
3:27:00So when we start saving money and we make the models cheaper to run and the agents cheaper to run, we end up passing those savings on to you.
3:27:07So we're really happy with where Kiro has gotten on that Pareto front of sort of cost performance and cost effectiveness.
3:27:13Excuse me.
3:27:13Now, like I said, we're going to spend this day talking about KiroCrew, and KiroCrew
3:27:17was built to run many Kiro agents at the same time.
3:27:20It's all open source.
3:27:21It actually started internally as an internal Amazon product where some engineers grabbed
3:27:27our CLI and they're like, hey, we want to extend this.
3:27:29It was during the whole OpenClaw craze, if anybody has their Mac minis at home, they've
3:27:34been running those.
3:27:34It happened during that craze where people wanted to figure out how they could get these
3:27:38agents to be more like personal assistants.
3:27:39And so this MeshClaw product internally was developed,
3:27:42got around 40,000 new internal Amazon developers on it
3:27:46in the first two months.
3:27:48And so us at the Kiro team were like,
3:27:49hey, since it was built on Kiro,
3:27:51we should take that and extend it and release it.
3:27:53And so we've done that now.
3:27:54And Kiro Crew is released.
3:27:56It's open source.
3:27:58And if I talk about what have we seen with the Kiro Crew team
3:28:02and our team in terms of building and developing this,
3:28:05we shipped on August 4th externally.
3:28:07And again, it's open source,
3:28:09so you can go look at all the source and look at all the PRs that came through.
3:28:12We go by one philosophy, that if we don't use it and love it, then we don't ship it.
3:28:17Since we launched, we've merged about 16,500 PRs with a very small team.
3:28:22Just last week, there was over 3,000 in a single week alone.
3:28:26Take contributions from external entities, both internal to Amazon and external to Amazon,
3:28:31people who just want to contribute and take a good idea with open source.
3:28:35And if you look at the chart here, we sort of leveled out at around, you know, if you
3:28:39look at just our shipping speed, we sort of leveled out at around high 100, 600 to 800,
3:28:45sort of the first few months after we shipped.
3:28:48And then you'll see in like 928, right, all of a sudden we break 1,000 again, and then
3:28:52we're sort of hovering around, I'll average it out in my head and say 1.2K-ish, right?
3:28:57And then you can see we made another step function improvement.
3:29:00And while I'm showing you PRs on this chart, what we really care about among the team is
3:29:04not necessarily the PRs, but we care about how many sessions can an individual developer
3:29:10operate at the same time.
3:29:12And I want to be clear that this is not a one size fits all thing, so as I walk through
3:29:16this, just keep that in mind that what works for us may not work for you, and so you should
3:29:20challenge what are you doing that you could learn from this.
3:29:25Last thing on this one, about one in ten of these is features.
3:29:28So I always get asked, like, man, 3,300 features,
3:29:31that's a lot of features in a week.
3:29:33Well, no, it's not all features, right?
3:29:34This is what shipping with speed looks like.
3:29:36A lot of, so I get asked, okay,
3:29:38so about one in 10 of these is a feature, so about 10%.
3:29:42And so if 10% are a feature, what are the other 90%?
3:29:46Well, for us, it's tests.
3:29:47Tests is a big part of what we do.
3:29:50It's, I'll say, paper cut fixes for things we've already shipped
3:29:53that we're just sort of learning about how it's getting used
3:29:56and we're massaging it a little bit.
3:29:58It's bug fixes as well.
3:29:59And that's
3:30:00really what shipping at a high speed with velocity looks like, right?
3:30:03It's not all new features, but if you just think about 10% of our features, there's still
3:30:06hundreds of features, new features being shipped over this timeline.
3:30:11If I think about sort of how did we evolve with our development cycle, we started, if
3:30:17I go back to when our CLI product initially launched, it was one session, one person.
3:30:21So I'm sure many of us experienced this
3:30:23when these new agents came out,
3:30:25and I was typing, and it would then work,
3:30:27and then it would stop working,
3:30:28and then I would type again, right?
3:30:29We'd just go back and forth.
3:30:31And then from that, so that was one session, right?
3:30:33I said we're trying to optimize sessions per person.
3:30:36Then we moved to parallel sessions,
3:30:37where instead of just running one session,
3:30:39I'm looking at, you know, maybe I'm running two or three,
3:30:41but I'm running them in the same manner, right?
3:30:43And the problem with both of these is that
3:30:45if you make yourself the feedback loop,
3:30:48you are now the bottleneck, right?
3:30:49These agents got faster long before you're going to be able to keep up.
3:30:53And so these first two sessions, they're just going to kill in one chart here,
3:30:56where it's, you know, we got to two or three sessions per person,
3:30:58but we sort of topped out.
3:31:00Now, stage three introduced memory and schedules.
3:31:03And with a memory, a new session now all of a sudden knows your conventions, right?
3:31:08So implicit memories.
3:31:09And then schedules make it so you don't have to kick all of the work off on your own.
3:31:13Things can happen on a schedule.
3:31:15The thing we learned here was that schedules only help if your machine is on.
3:31:19So in the previous phases, most of our engineers were running locally, right, as I'm sure many of you do still today.
3:31:24But moving to remote ends up making it so you no longer have to worry about, is it going to run?
3:31:28Because I have an always-on machine, something we're very good at at AWS.
3:31:32So obviously we are running remote.
3:31:35But what happened then is this allowed us to sort of, I'll say, build up to between 10 and 20 sessions, right?
3:31:40And then we sort of plateaued again.
3:31:42We were unable to sort of break that barrier with just memory and schedules.
3:31:46because much of what we were doing
3:31:48was still being the glue between the sessions, right?
3:31:52And then if we, our next phase was a pipeline,
3:31:55but not a CICD pipeline like you might be thinking,
3:31:57but instead more like an assembly line.
3:32:00I actually have, my previous life, I worked in aerospace,
3:32:03so a lot of assembly lines,
3:32:04and the hard thing with assembly lines is nonstandard work, right?
3:32:07So if everything is exactly the same as it flows through the system,
3:32:10an assembly line works great.
3:32:11As soon as you get nonstandard work,
3:32:13all of a sudden it becomes difficult,
3:32:14and I'm sure all of you know this
3:32:16but software doesn't follow the exact same
3:32:18path every time
3:32:19and so what we found our own teams doing then
3:32:22was this, sorry, this got us to maybe like
3:32:2430, 20, 30, 40
3:32:26sessions-ish but what happened
3:32:28was is our team was then trying to
3:32:30unblock the assembly line or the pipeline, right
3:32:32so as things came up if we
3:32:34thought something was a small issue and then it turned into
3:32:36a big issue, sometimes the assembly line would be
3:32:38stuck, somebody would have to come in and unblock
3:32:40that thing, maybe you restart it, maybe you replan
3:32:42it. And so this is only going to carry us so far because, again, we are in the midst
3:32:47of the agent. We're blocking the agent more than the agent wants to work.
3:32:53And so what did we do for that? Well, we launched a thing called Ultra Mode. Ultra Mode
3:32:58is still, just to be clear, if you go try it and you look for it, it's still in the developer
3:33:03preview, so it's behind a feature flag. But because it's open source, the code is all
3:33:06there. If you want to go try it, you can try it out. As I said up front, if we don't love
3:33:10it, we don't ship it, so we're still sort of finishing, putting the finishing touches
3:33:13on this one.
3:33:15But I wanted to talk about it anyway, because it takes you from being, what we did was,
3:33:19if you look at this middle part here, where we are the person driving the effort, these
3:33:24are the things that we might have been doing.
3:33:26So watching for problems was one of the biggest ones, and then trying to fix the assembly
3:33:29line when the problem came up.
3:33:31Maybe you're judging a result, or you're deciding what to do next.
3:33:35What Ultra Mode does is it does that job for you, right?
3:33:38So you put an agent on top of your agents that then is responsible for doing that, defining
3:33:42the goal based on whatever you're trying to accomplish, starting things, watching for
3:33:46problems, unsticking things.
3:33:48And this is getting us today to 50 plus sessions.
3:33:52And so if you think about where we started with a single session and a single person,
3:33:56now if a single developer is doing 50 plus sessions, that's how you get that sort of
3:34:00the volume in the PR chart I showed you earlier.
3:34:04And we can already see the next wall that's coming.
3:34:07The orchestrators working with other orchestrators,
3:34:10and then the governance around what you want that to be
3:34:12and the guardrails become even more important, right?
3:34:16So again, that's how we got the volume.
3:34:18Those were the five phases we went through.
3:34:20Ultra mode has been a game changer for us,
3:34:22as was the assembly line.
3:34:23And I invite you to try those ideas
3:34:25if you think they might work for you.
3:34:27The next thing I wanted to talk about, though,
3:34:29was what did we learn from this?
3:34:31Like, what are the biggest things that prevented us from shipping quickly?
3:34:35The first one is test everything.
3:34:38And now this is no, this is not a new lesson, right?
3:34:40Everybody in software knows this.
3:34:41We should be doing this.
3:34:43But I still see people too often discount the value this is providing.
3:34:48And this is gonna build in with the next one I have behind this around reviews.
3:34:52Like, if you think about where do you get confidence to ship something?
3:34:54I think many people still feel like, hey, I get confidence because I looked at all the
3:34:58code and I know exactly what that code does. Again, if you do that, like maybe that's a
3:35:03good thing, but where we have gotten our confidence, and Amazon's always been very test-driven,
3:35:08is from the tests. Because you might not be able to look at something every time, but
3:35:12you can certainly test something every time in the exact same way and test the result,
3:35:17right? And come out with the result. And so for us, again, quality really becomes a testing
3:35:22problem and we're spending a significant portion of our time, around 20% just on testing today.
3:35:29The other big thing we learned, and this is a bigger one, and also a contentious topic,
3:35:33is, is reviews.
3:35:35So quick show of hands, how many people have done a review with an agent, thinking that
3:35:39there's going to be like one or two comments, and then you come back and you're like, oh
3:35:43my god, like this is terrible code, right?
3:35:47There's comments everywhere.
3:35:48We had the same problem.
3:35:49So we originally wrote a review agent that would review everything, and our goal was
3:35:53to fix every single comment.
3:35:54Well, as you guys, I heard the laughter, so as you expect, you submit changes, so you
3:36:00submit a PR, you get the feedback on that PR, you come back, you submit it again after you
3:36:05fix it, you come back and now it's three new things that weren't even tested, weren't
3:36:08even told about before and it's four things on the things you already fixed.
3:36:11You submit it again and this cycle just continues to repeat.
3:36:14So what we've done with reviews is now we categorize the reviews or triage them into
3:36:19blocking and non-blocking issues.
3:36:21So, for example, something that breaks the build,
3:36:25if there's something obviously that breaks the build,
3:36:26that's a blocking thing,
3:36:27and we need to go fix that before we merge this PR.
3:36:30Changing a variable name, not a big deal.
3:36:32We'll take care of that later,
3:36:33so we let those things slide through.
3:36:36The other thing that we've done with reviews
3:36:38is we want to have fast feedback for an agent,
3:36:40and as you can see by the process here
3:36:42that I'll walk you through in a second,
3:36:43this is not that fast.
3:36:44And so we have a fast gate
3:36:45that really double-checks things really quickly
3:36:47to make sure that things are in the right folders
3:36:49before we move on to our reviews.
3:36:51We have six reviewers, actually, that run.
3:36:53Two code reviewers, both in different models,
3:36:56with different model families.
3:36:58And the intent there is to really find,
3:37:00to make sure you don't have a model gap, right,
3:37:02so that the model wasn't trained on something that we don't catch.
3:37:04This is where we use the Anthropic and OpenAI models
3:37:07or an open weight model to be those levers.
3:37:10We run a design review that asks,
3:37:13does this thing have the right shape?
3:37:14Meaning, is what I'm building,
3:37:16like if I'm adding another cache in this PR,
3:37:18can the cache I already have take care of this issue or not.
3:37:22We have a first principles review that says,
3:37:24is this change addressing the real problem?
3:37:27We have a UX review as well,
3:37:29that if you're making it, let's say you're creating a new error message,
3:37:32we have an agent decide whether or not that error message,
3:37:34the human will know what to do with that error message
3:37:36based on what we've published.
3:37:38And then we're consistently tightening our security posture.
3:37:41Now, each one of these reviewers gets one vote.
3:37:43It's either yes or no. It's red or it's green.
3:37:45If everything's green, it goes on to merge.
3:37:47If not, we go to a prepare PR skill that's in the repo,
3:37:50so you can check the skill out if you want.
3:37:51And that prepare PR skill then tries to resolve the issues as we go.
3:37:57The third lesson, and sorry, on the reviews, one more thing.
3:38:00Like I said, we've stopped reviewing code a long time ago.
3:38:03We have moved on, and what we're responsible for is not the code itself.
3:38:06See, this actually goes back. It does.
3:38:08What we're responsible for is the review record.
3:38:10Did it follow the process as we expected?
3:38:12And did we get the results we wanted?
3:38:14If not, we fixed the review process.
3:38:17Two other things. So, one, the over-engineering problem. So we have a specific role to teach
3:38:24our agents to subtract, not just add. And so we try to remove code from the process
3:38:28as we go to prevent that over-engineering. And we use that same philosophy on our roadmap
3:38:32as well, where saying yes is becoming very cheap or has become very cheap, right? It's
3:38:36very easy for us now with code to say, like, yes, I'll do that and I'll do all of these
3:38:40things. Whereas we used to have to be very particular about what we were going to build.
3:38:44The problem with saying yes all the time
3:38:46is that you might lose your product shape,
3:38:48and we're already seeing this in KuroKribit,
3:38:50that it's moving so fast and so quickly
3:38:52that you don't want your users to get caught in the noise.
3:38:55And so, again, use the same principle on the roadmap
3:38:57as what we do.
3:38:59And then last thing on lessons is
3:39:01you're not done once you merge a PR, right?
3:39:04There's still other phases,
3:39:05and so for us, merging is not the last check that we do.
3:39:08Instead, we run a release gate
3:39:09where the agent will download the repo,
3:39:12It installs it, runs it, and then checks the core flows
3:39:14to make sure that the overall system is still in good shape
3:39:17before we would do a release.
3:39:18And then we do security all the time as well.
3:39:21So we have every hour, every day, an always-on,
3:39:23every hour against our security guidance,
3:39:25every day against an AppSec review,
3:39:27and then an always-on pen test agent.
3:39:30So again, merging is not the end of the process.
3:39:32You have to crunch everything down.
3:39:35So I'm going to come back to where I started.
3:39:37Are you getting everything from your agent?
3:39:39and many of us don't feel like we are yet.
3:39:42And what I'm going to offer here today,
3:39:43or what I've tried to offer is,
3:39:45the agent was fast enough to work on this a long time ago
3:39:49and it's only getting better and better and better.
3:39:51And so what we have been doing is challenging how we work
3:39:54and figuring out where are we preventing the agent
3:39:56from working as quickly as possible.
3:39:58And I've given you numerous
3:40:00examples today.
3:40:01Again, I'll just call it a couple of them here.
3:40:03So it started with our attention span, right?
3:40:05When I was the feedback loop, I could no longer ship because I was the feedback loop.
3:40:09When I wasn't there, the agent stopped working.
3:40:11The laptop lid closing.
3:40:12All of a sudden, you lose your connection.
3:40:14Moving to remote can solve that, right?
3:40:16The fixed pipeline with a fixed shape that can't respond when things go slightly off path.
3:40:21Again, how do you fix that?
3:40:22We fixed it with an orchestrator.
3:40:24ultra-mode. The endless
3:40:26reviews that I think I've gone into probably
3:40:28more depth than you were interested in, but
3:40:29the endless review cycle, and then the loop that only
3:40:32adds, right? The loop that only adds things
3:40:34to it and doesn't try to subtract.
3:40:38And so
3:40:38I'll leave you with, I also want to be
3:40:40clear that this is going to be a
3:40:43continual adjustment,
3:40:44right? This is not, like, this may be what works for us
3:40:46today. I promise you another month we're going to find
3:40:48something else that works, and we're going to continue to evolve
3:40:50significantly. But like I said in the
3:40:52beginning, the pace that we're finally able to deliver at, I feel like for us, is that finally
3:40:57we're meeting that promise of AI, and I still see the disparity among other teams.
3:41:03And so, like I said, I'm going to leave you with this. If you're interested in trying Kiro Crew and
3:41:08building your own software factory, scan this QR code. It's a workshop. We have a pretty generous
3:41:13free trial on Kiro, so you can go sign up for a free trial, go install Kiro Crew, and you'll be
3:41:19off and running.
3:41:21And if you don't, if you want to try, like I said, if you want to try CreoCure, you can
3:41:23do that.
3:41:24The same free trial works across all of the products.
3:41:26So the IDE, the CLI, the web client, the mobile client, try them all with that free trial.
3:41:31And with that, I'm excited to see what you all build.
3:41:34I appreciate the time.
3:41:34Thank you.
3:41:35Enjoy the rest of the conference.
