Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro

Talk transcript

1. 0:12

   [upbeat music] For those of you who haven't heard of us, Kiro is an agentic IDE. Um, we launched generally available this most recent Monday, I think the seventeenth, but we launched public preview on, uh, in July, uh, I think July fourteenth.

2. 0:34

   So out there for a few months, getting customer feedback, um, all that good stuff. We're gonna talk a little bit about using spectrum and development to sharpen your AI toolbox.

3. 0:42

   I did a show of hands. About a quarter of the people here are familiar with spectrum and dev. My name's Al Harris, um, Principal Engineer at Amazon. I've been working on Kiro for the last year.

4. 0:50

   Uh, and we're a very small team. We were basically [REDACTED:generic_id] or four people sitting in a closet doing what we thought we could do to improve, um, the software development life cycle for customers.

5. 1:00

   So we were cha-- we were charged with building a development tool that's-- that answered, um, that improved the experience for spectrum and development. We were theoretically funded out of the org that supported things like Q Dev, but we were purposefully a very different product suite from the QE system to just take a different take on these things.

6. 1:18

   So we wanted to work on scaling, you know, helping you scale AI dev to more complex problems, uh, improve the amount of control you have over AI agents, and improve the code quality, and maintain, uh, reliability, I should say, of what you got out the other end of the pipe.

7. 1:32

   Now we're back to new content. Um, so our solution was spect driven. We took a look at some existing stuff out there and said, "Hey, vibe coding is great," but vibe coding relies a lot on me as the operator getting things right.

8. 1:43

   That is me giving guardrails to the system. That is me, uh, putting the agent through a s-- uh, kind of a strict workflow. We wanted spect driven dev to sort of represent the holistic SDLC because we've got, you know, twenty-five, thirty years of industry experience, um, building, uh, software, building it well and building it with different practices,

9. 2:02

   right? We've gone through Waterfall, Agile, XP. Um, we have all these different ways that we represent what a system should do, and we wanna effectively respect what came before.

10. 2:12

    So, uh, this animation looked a lot better. It was initially just the left diamond. But I-- the idea was, hey, you know, you basically are iterating on an idea.

11. 2:22

    I think like half of software development is discovery requirements. Um, and that discovery doesn't just happen by sitting there and thinking about what, what should the system do? What can the system do?

12. 2:31

    We, we realized though, kind of working on this, that the best way to make these systems work is to actually synthesize the output and be able to feed that back really quickly.

13. 2:39

    Things like your input requirements, um, to actually do the design and feed that, you know, realize, oh, actually, if we do this, there's a side effect here we didn't consider.

14. 2:46

    We need to feed that back to the input requirements. And so this compression of the SDLC evolved to bring structure into the software development flow. We wanted to take, um, the artifacts that you generate as part of a design.

15. 3:00

    That's the requirements that maybe a product manager or developer writes. That's gonna be the acceptance criteria. What does success look like at the end of this? And then we wanna take the design artifacts that you might review with your dev team, you might review with, you know, stakeholders and say, "This is what we're going to go build

16. 3:14

    and implement the thing." And we wanna make sure that you can do this all in some tight inner loop. Um, and also that was initially what spec-driven dev was.

17. 3:23

    Um, what spect driven development in Kiro is today, or at least was before we went GA, was, uh, you give us a prompt, and we will take that and turn it into a set of clear requirements with acceptance criteria.

18. 3:36

    We represent these acceptance criteria in the EARS format. EARS stands for the Easy Approach to Requirement Syntax. Um, and this lets you really easily, uh, it's effectively a structured natural language representation of what we-- you want the system to do.

19. 3:50

    Now, for the first four and a half months this product existed, the EARS format looked like kind of an interesting decision we made, but just that sort of interesting.

20. 3:58

    Um, and with our launch, our general availability launch on Monday, we have finally started to roll out some of the side effects of that, which is property-based testing. Um, so now your EARS requirements can be translated directly into properties of the system, which are effectively invariants that you wanna deliver.

21. 4:14

    Um, for those of you who have or, like, have not, I guess, done property-based testing in the past using something like, I think it's, uh, Hypothesis in Python or FastCheck in Node, um, Closure's Spec library is another example.

22. 4:30

    These are, uh, approaches to testing your software system where you're effectively trying to produce a single, uh, test case that, that falsifies the invariant that you wanna prove. And if you can find any, uh, contrapositive, then you can say this requirement is not met.

23. 4:46

    If you cannot, you have some high degree of confidence where the word high there is doing a little bit of heavy lifting because it depends on how well you write your tests.

24. 4:54

    But you can say with a high degree of confidence that the system does exactly what you're saying it does. Um, yeah. So a property-- We, we'll get a little bit more into property-based testing and PBTs a little later.

25. 5:09

    But this is the first step of many we're taking to actually take these structured natural language requirements and then tie this with a through line all the way to the finished code and say, if your code, if the properties of the code meet the initial requirements, we have a high degree of confidence that you have re-- uh,

26. 5:27

    reliably shipped the, the software you expected to ship. So with spect driven dev, we take your prompt, we turn it into requirements, we pull a design out of that, we define properties of the system, and then we build a task list, and we go, and you can run your task list.

27. 5:44

    Effectively, the spec then becomes the natural language representation of your system. It has constraints, it has concerns, um, around functional requirements, non-functional requirements, and it's this set of artifacts, uh, that you're delivering.

28. 5:57

    So I don't think I have the slide in this deck, but ultimately, the way I look at spec is that it is, one, a set of artifacts that represent sort of the state of your system at a point in time T.

29. 6:06

    It is, two, a structured workflow that we push you through to reliably deliver high-quality software, and that is the requirements, design, um, and execution phases. And then [REDACTED:generic_id], it is a set of tools and, and, um, systems on top of that that help us deliver reproducible results where one example of that is property-based testing.

30. 6:25

    Another example of that, which is a little less obvious, but we can talk about later, is going to be... Um, I don't even know what to call it. Uh, requirements verification.

31. 6:34

    So we scan your requirements for over-ambiguity. We scan your requirements for, um,

32. 6:40

    invalid constraints, e.g., you have, uh, you have conflicting requirements, and we help you resolve those ambiguities using sort of classic, uh, automated reasoning techniques. Um, and I could talk a little bit more about sort of the, the features of Kiro.

33. 6:55

    I think that's maybe less interesting for this talk 'cause we wanna talk about spectrum dev. We have all the stuff you would expect, though. We have steering, which is sort of memory and sort of cursor rules.

34. 7:04

    We have MCP integration. We have, you know, image, yada, yada. Um, so we have ways to... And we have software hooks. Um, so let's talk a little bit about sharpening your tool chain, and I'm gonna take a break really quick here.

35. 7:17

    Uh, just pause for a moment for folks in the room who had maybe tried downloading Kiro, um, or something else, and just say, are there any questions right now before we dive into how to actually use Spec to achieve a goal?

36. 7:32

    No questions. It could be a good sign. Could mean I'm not, uh, talking about anything that's particularly interesting. So, um, I actually wanna, like, talk in some concrete detail here.

37. 7:43

    Uh, this is a talk I gave a few months ago on how to use MCPs in Kiro. And so one of the challenges that people who had tested out Kiro had, that might be a little easier to see, was that they,

38. 7:56

    um, they felt that the flow we were pushing them through was a little bit too structured. Like, you don't have access to external data. You don't have access to the, to all these other things you want.

39. 8:06

    And so one thing that we said on our sort of journey here towards opening your, um... Oh, you know what? It's out of order. Here's my nice AI-generated image.

40. 8:15

    So you can use MCP. Everybody here, I assume, is familiar with MCP at this point. But, uh, Kiro integrates MCP the same way all the other tools do. Uh, but what I think people don't do enough is use their MCPs when they're building their specs.

41. 8:30

    And so you can use your MCP servers in any phase of the spectrum and development workflow. That's gonna be requirements generation, design, um, and implementation. Um, and you can use...

42. 8:42

    We, we'll go through an example of each. So first of all, to set up a spec in Kiro is fairly straightforward. We have the Kiro panel here, which is a little ghosty.

43. 8:52

    Um, and then you can go down to your MCP servers and click the plus button. You can also just... My favorite way to do it is to ask Kiro to add an MCP, uh, and then give it some, some information on where it is, and it can go figure it out usually from there.

44. 9:06

    Or you just give it the JSON blob, and it'll figure it out. Once you have your MCP added, you'll see it in the control panel down here, and you can enable it, disable it, allow list tools, disable tools, et cetera.

45. 9:16

    So you can manage context that way. Worth noting, changing MCP and changing tools in general is a cache-breaking operation, so if you're very deep into a long session, maybe don't tweak your MCP config 'cause it will slow you down dramatically.

46. 9:29

    But let's talk about, um, MCP and spec generation. So something I, the Kiro team uses Asana, um, for reasons I don't know, but it's our task tracker of choice.

47. 9:42

    Uh, but so one thing I wanna do is, uh, maybe go and say, "I don't wanna write the requirements for a spec from scratch. My product team has already done some thinking.

48. 9:49

    We've iterated in Asana to kinda break a project down." This is not always how things work, but sometimes how things work. So in this case, I have, I have a task in Asana...

49. 9:58

    Oh, no. I did the wrong thing. That's what I get for zooming. So I have this task in Asa- in Asana that says, "Add the view model and controller to this API."

50. 10:09

    In this case, this was a particular demo app that I can share in a few minutes, and we even had like, it's kind of peaking under here, but we had some details about what we wanted to have happen.

51. 10:19

    Now, I can go into Kiro and just say, "Start executing task XYZ URL from Asana." And Kiro's gonna recognize this is an Asana URL. I have the Asana MCP installed.

52. 10:29

    It goes and pulls down all the metadata there, um, da, da, da. So it's going to break out and from there start, um,

53. 10:36

    start determining what to work on. Um, oh, it's funny. These titles are backwards.

54. 10:46

    Basically, create a spec for my open Asana tasks. Again, go pull from Asana all the tasks, and then for each one, generate, um, requirements based on those tasks. So I think I had, like, six tasks assigned to me.

55. 10:57

    One is do user management, do some sort of, um,

56. 11:04

    uh, property management, da, da, da. It pulled them in, generated the requirements, and then in this case... Title is wrong. Apologies. Start executing task. This is, I wanna go and do the code synthesis for this.

57. 11:16

    Um, and I will take a quick break here to talk about how you can do this in practice. So for those of you who are, you know, following along in room, uh, feel free to fire up your Kiro, open a project, and then pick an, uh, an MCP server.

58. 11:30

    I'll share a few repos here really quick that you can play around with.

59. 11:35

    So I have an MCP server implemented. I have,

60. 11:44

    hmm, this Lofty Views, which I think implements the Asana. Um, and then these should all be public. Let me just double-check.

61. 11:57

    Yeah, okay. So for example, if you wanted to extend my... I have a Nobel Prize MCP, which curls. Gramps, unsurprisingly, there is a Nobel Prize API, um, so you can use UVX to install it, or you can Git clone this Al Harris at Nobel MCP.

62. 12:13

    Uh, this is just one example. Another one here is if you wanna play around with the sample that's in the video, um, I have Al Harris at slash lofty views.

63. 12:21

    Um, I'll leave these both sort of up on the screen for a few moments for folks who do wanna copy the, uh, the URLs.

64. 12:31

    But while that is happening... Oh, no. Let's put you on the same window[clears throat]

65. 12:48

    So what I'll demo quick is the usage of an MCP to make, like, spec generation much easier or more reliable. So here I have... Let's see.

66. 13:01

    Got a lot of MCPs. Which ones do I actually wanna use?

67. 13:11

    Let's use the GitHub MCP. Oh, no. Ignore me.

68. 13:22

    That's better. Okay, well, I have the Fetch MCP. So in this case, I could, for example, come in here and say, "Hey, I've generated a bunch of tasks here, this lofty views app."

69. 13:33

    This is basically a very simple CRUD web app. Um, but I want Kiro to,

70. 13:41

    uh, use the Fetch MCP to pull examples from similar products that exist on the internet. You could also use, you know, Brave Search or Tavily Search MCP servers, but in this case, I'll just use Fetch 'cause I've got it enabled.

71. 13:54

    Um, so let's say... Oh, actually we can run the web server and use Fetch. That's a good example.

72. 14:21

    This is one example of, you can, at any point in the workflow generating a spec, go through and, um,

73. 14:26

    you know, use your MCP servers to get things working. Oh, no. This is what I get for not using a project in a while.

74. 14:35

    Uh, we'll cancel that. We can actually do something a little more interesting, which is a separate project I've been working on. Um, so I've been working on a, an AgentCore agent, and that might be...

75. 14:47

    I, I know the project works, which is the reason I'll fire it up here. What did I call it?

76. 15:04

    Well, maybe we'll do live demos at the end.

77. 15:07

    So that's sort of like the most basic thing you can do with Kiro is just use MCP servers. But any tool uses MCP servers. I actually don't think that's particularly interesting.

78. 15:16

    So let's say in sort of this process of trying to sharpen our, our spec-driven dev toolkit, we've finished up with the two hundred grit. We've added some capabilities with MCP.

79. 15:25

    It's useful, but it's not gonna be a game changer for us. I wanna come in here and actually get it up to the four hundred grit. Let's get, start to get a really good polish on this thing.

80. 15:33

    I wanna customize the artifacts produced because you've got this task list, you've got this requirements list, and I don't agree with what you put in there, Al. Um, you could say that, a lot of people do, and I...

81. 15:43

    That's a, a great starting point. So here's something I heard earlier in the week at, um, uh, you know, earlier in the conference, is that people like to do things like use wireframes in their mocks.

82. 15:54

    Um, use wireframe mocks because in your specs are natural language, using specs as a control surface to explain what you want the system to do. Uh, therefore, I wanna be able to actually put UI mocks in here.

83. 16:05

    So the trivial case is that I just come in here and say, Kiro's asked me here, "Does, does the design look good? Are you happy?" And I said, "This looks great, but could you include wireframe diagrams and ASCII for the screens we're going to build here?"

84. 16:17

    I'm adding... This is again from that lofty views thing. I'm adding a user management UI. But I wanna actually see what we're sort of proposing building, not just the architecture of the thing.

85. 16:27

    So Kiro's gonna sit here and churn for a few seconds, but you can add whatever you want to any of these artifacts because they're natural language. So they're structured, which means we want some re, re, um, some sort of, uh, reproducibility in what they look like, but ultimately, what they look like doesn't matter 'cause we've got the,

86. 16:43

    the any machine here, the agent sitting that can help translate it to what it needs to be. So Kiro's churning away here. It's thinking, thinking, and then it's gonna spit out these, uh, text-wrapped ASCII diagrams.

87. 16:55

    I'll fix the wrapping here in a second in the video. But ultimately, like,

88. 17:01

    you know, it does whatever you want. So if you want additional data in your requirements, you can do that. If you want additional data in the design like this, uh, you can easily add that.

89. 17:13

    Here we've got sort of these wireframes in ASCII that help me sort of rationalize what we're actually about to ship. Um, and then I can again continue to chat and say, "Actually, in the design, I don't want, um, you know, maybe I don't want this Add User button to be up at the top the entire time," in

90. 17:27

    which case I could chat with it to make that change easily. And now we're on the same page up front instead of later during implementation time. So we've again sort of left shifted some of the concerns.

91. 17:37

    Um, so that's one example. You know, I wanna add UI mocks to the design of a system. Another example, though, could be this, um... Oh, this is a just quick snapshot of the end state there, where now my design does have these UI mocks.

92. 17:51

    Um, but another example that I actually like a little bit more is this, uh, including test cases in the definition and tasks. So today, the tasks that Kiro will give you will be, will be kind of the bullet points of the requirements and the acceptance criteria you need to hit.

93. 18:05

    But I wanna know that at the end state of this task being executed, we have a really crisp understanding that it is correct. It's not just, like, done. 'Cause the a-- anybody who's used an agent can probably testify that, um, the LMS are very good at saying, "I'm done.

94. 18:20

    I'm happy. I'm sure you're happy. I'm just gonna be complete." Oh, the... Yeah, the tests don't pass, but they're annoying. I tried [REDACTED:generic_id] times to get them to work.

95. 18:26

    I'm just gonna move on. Um, no, I don't want that. I wanna actually know that things are working. So in this case, I've asked Kiro to, um, include explicit unit test cases that are going to be covered.

96. 18:36

    So my task here, for example, in create, creating this AgentCore memory check pointer is going to have all the test cases that need to pass before it's complete, and then I can use things like agent hooks to ensure those are correct.

97. 18:47

    We'll run this, uh, sample a little later in the talk. Um, this is the thing I'm ready to demo

98. 18:53

    Uh, yeah, so this is another example where you can-- Again, you're, you're working on your tool bench, you're sort of... You have all these capabilities and primitives at your control, and you can tweak the process to work for you, not just the process that I think is the best one.

99. 19:08

    And then sort of last but not least, the eight hundred grit. At this point, we're getting a final polish on the tool. Uh, we might be stropping next, but we wanna, y-you know, you can iterate on your artifacts, but you can also iterate on the actual process that runs.

100. 19:22

     So one thing you might have, and I do this a lot, is I'll, I'll be chatting with Kiro and I say, "Hey, I want to, um..." In this case, I wanna add memory to my agent in AgentCore.

101. 19:34

     Um, let's dump conversations to an S3 file at the end of every execution. Kiro's gonna say, "That's great. I know how to do that. I'm gonna research exactly how to do that thing.

102. 19:43

     I will achieve this goal for you." But ultimately, what I've done is actually introduce a bias up front, which is I'm steering the whole agent. It's using S3 as the storage solution just because maybe I'm familiar with it, but it's probably not the best way to go about it.

103. 19:57

     So then after it had synthesized the design and all the tasks and all this stuff, I came back and said, "Well, like, we don't need to stick to this rigid specter of dev workflow that I've-- that has been defined by Kiro."

104. 20:07

     I can ask for alternatives. Like, is this the idiomatic way to achieve session persistence? I-I don't know. Maybe there's a better way. Maybe if we're talking AWS services, S3, it's Dynamo or yada yada.

105. 20:18

     Uh, Kiro's gonna come in here and say, you know, "Good question. Uh, da, da, da. Let me research." It's gonna go through, call a bunch of MCP tools that I've given it access to.

106. 20:27

     This kind of ties back to that you should be using MCP. And then it comes back with this recommendation that I didn't know was a feature, which is AgentCore memory.

107. 20:36

     Um, it says it's more idiomatic and future-proof. That maybe is TBD and should be checked a little closer. Um, but, uh... Or you could use S3, which is the thing you recommend.

108. 20:47

     Now, actually, I, I bet there's far more than two options here. So you could probably keep asking the agent, are there other options, yada yada, and it would go and continue to investigate.

109. 20:55

     But you should not lock yourself into the rigid flow that is sort of the starting point here. Um, yeah. So that, that's actually, I think, it for my deck.

110. 21:04

     Um, what I will talk about is let's just run through that sample I just had up there, which is that, um...

111. 21:15

     So basically, let me delete, delete it, and I'll just do a live demo of sort of specs in Kiro and how we can fine-tune things a little bit. So this project is a Node.js app.

112. 21:29

     It is a, um, it's a CDK... A-again, I'm not trying to sell more AWS. This is just the technologies I'm familiar with, so I can move a lot more quickly.

113. 21:39

     So I wanted to know a little bit about AgentCore, which is a new AWS offering. And as somebody building an agent, I should probably be familiar with it, so.

114. 21:46

     And I'm not familiar enough with it, so I've got-- we've got some other people here who know a lot about it, so gonna put my hand up a little bit and, you know, you caught me.

115. 21:54

     So I set up a CDK stack, which is just, um, you know, IAC technology to deploy software. I'm familiar with it and I love it. Uh, so I have a stack here that lets me deploy whatever an AgentCore runtime is.

116. 22:07

     I don't know. I asked Kiro to do it. We vibe coded this part. So we vibe coded the general structure. We got an agent. We got IAC set up.

117. 22:14

     I then vibe code added CommitLint. I added Husky, a few things like this that I like for my own TypeScript projects, um, Prettier and ESLint, I think. So we've a basic product here, or like a basic project here that I know I can deploy to my personal AWS account.

118. 22:29

     Um, now I'm gonna come in here and... Oh, and then importantly, this is super important because I don't know how the hell AgentCore works. And I could go read the docs, but the docs are long and they're complicated, and I'm really just trying to build out a POC to, to, like, learn about it myself.

119. 22:44

     So I added two MCP servers. Oh, no, maybe I didn't. Let me check. Oh, okay. Yes, sorry. Buried down here at the bottom. So this is my Kiro MCP config.

120. 22:59

     I added one important MCP server here, which is the AWS documentation one. There's other ways to get documentation. You can use things like, um, Tessle level seven. But in this case, this is vended by AWS, so I have some confidence that it might be correct.

121. 23:13

     So I used this to help the agent have knowledge about sort of what technologies exist, and I think I used Fetch quite a bit as well. So these are the two sets of, um...

122. 23:24

     These are the two steps-- sets of, uh, MCP servers I provided the system. That's great. Move on.

123. 23:32

     Um, so... And I'll just rerun this from scratch. So what I had done yesterday evening or maybe the evening before was I sat down and I have this system sort of basically working, and now I wanna start doing specter-driven development.

124. 23:48

     So I wanna add this, uh, session ID concept, and then I wanna read/write conversation to an S3 file, blah, blah, blah. This is the whole sort of bias thing I showed you earlier.

125. 23:58

     We're gonna fire that off through Kiro. It's gonna start running, uh, chugging away, and then it's gonna, you know, see if the spec exists. Uh, okay, the folder does exist.

126. 24:07

     It's probably gonna realize there's no files there and start working away. But, um, from here, I'll sort of live demo. It's gonna read through require-- It's gonna read through existing docs.

127. 24:17

     It's gonna read through existing files, gather the context it needs, sure, in a way. Um,

128. 24:23

     but in a moment, once it generates sort of the initial requirements and design, I am gonna challenge it to use its own, you know, MCP servers. I want you to go and do some research on the best way to do this and provide me some proposals.

129. 24:35

     Um, and this is why I was hoping to get the clip-on mic working, 'cause I've gotta set this down for a moment.

130. 25:05

     Okay. So, you know, I don't know if this is the best way to do this. Um, go read docs, go use Fetch, da, da, da. It's gonna keep kinda churning away here, and then come back to me after it's probably got a few ideas and proposed it.

131. 25:17

     But, um, this is an example of me just using additional capabilities. Uh, use Fetch, use the docs MCP, use whatever you can to get the best information, and don't take at face value the things that I said.

132. 25:30

     These are usually things we have to prompt pretty hard to get the agent to do, but if you're doing it in real time, it works fairly well. Um, again, the agent...

133. 25:37

     All of these agents are gonna be very easy to please, so, you know, just 'cause I said something in the student docs, it may or may not actually be the most important thing from the agent's perspective down the road.

134. 25:48

     So, you know. Okay, so it's done a little bit of research. It understands the LangGraph, which is the agent framework we're using, already has this knowledge of persistence. Um, da, da, da.

135. 26:00

     And actually, in this case, it didn't find... It did not use the MCP for, uh, AgentCore docs. It didn't find that A- AgentCore has this knowledge of persistence. Um, so maybe, like, let's assume I don't...

136. 26:11

     I still don't know that exists because I didn't dry run this a few days ago. Um, we might have to find that later in the design phase. So the first thing it's gonna do is kind of iterate over all my requirements here.

137. 26:21

     Um, you know, change the requirements based on what it now knows about LangGraph and how it can natively integrate with the, uh, checkpointing, but it's still really crisply bound to this, like, S3 decision that I made implicitly in the ask.

138. 26:34

     Um, so that is just something to be aware of. Any- anything you put in the prompt is effectively grounding the agent, um, for better or for worse. I see it's still iterating.

139. 26:45

     So yeah, comes through, says, "Does this look good?" We change da, da, da. I'm gonna say, "Looks great. Let's go to the design phase." So now Kiro's gonna take my requirements and take me into the design phase of this project.

140. 26:55

     I can make this so things are a little bit bigger.

141. 27:00

     But, um, here's an example of what I meant by these EARS requirements. So the user story here is, as a dev, I wanna implement a custom S3-based checkpoint, or so the agent can use LangGraph's native persistence mechanism with S3.

142. 27:13

     Great. That sounds reasonable to me as a person, you know, sort of co-authoring these requirements. This here, this sort of when-then-shall syntax, this is the EARS format, and the structured natural language is really important for us to pass this through non-LLM based models and give you more deterministic results when we parse out your requirements.

143. 27:33

     'Cause ultimately, our goal is to actually use the LLM for as little... Not as little as possible, but less and less over time. We wanna use classic, uh, automated reasoning techniques to give you high-quality results, not just, you know, whatever the latest model is gonna tell you.

144. 27:45

     Um, so Kiro's gone through, spits out a design doc. Let's actually just look at this in markdown.

145. 27:53

     This... Sure, you got a server, da, da, da. Checkpoint, or it goes to S3. That makes sense. Pseudocode. Again, in a real scenario, maybe I read this a little bit more closely.

146. 28:05

     And what's actually... This is the new thing we shipped in, um, on the seventeenth, is that now Kiro's going to go through and do this formalizing requirements for correctness properties.

147. 28:15

     Um, and so right now, what the system is doing is it's taking a look at those requirements you generated, uh, the requirements we agreed upon with the system earlier.

148. 28:23

     These look good. I agree with them, yada, yada. It's taking a look at the design, and it's extracting correctness properties about the system that we wanna run property-based testing for down the road.

149. 28:32

     This is something that may or may not matter for you in the prototyping phase but should matter for you significantly when you're going to production. Because if these properties are correct and these properties are all met, the system aligns one-to-one with the input requirements you provided.

150. 28:46

     Um, yeah. So while this is chugging away, any questions yet? Any folks kinda

151. 28:53

     curious about this?

152. 28:55

     Um, yeah.

153. 28:56

     Oh. We're hearing them there.

154. 28:59

     Um, what would you say is the main difference between planning mode that Cursor has?

155. 29:05

     Uh, I haven't used the planning mode in a couple of weeks, so it's... I'm... Things move so fast, it's a little wild. Um, but I think ultimately, uh, what we would say is that Kiro's spectrum in dev is not just LLM driven, but it is actually driven by, like, a structured system.

156. 29:21

     Um, and so planning mode, I'm not sure if there's actually, like, a workflow behind it that takes you through things. But, um, yeah, this is our take on it, for sure.

157. 29:31

     I'm not familiar enough to give, like, a more concrete example, unfortunately.

158. 29:34

     Well, pretty similar. I mean, it doesn't give you, like, this... I think that this tool is cool.

159. 29:38

     Mm-hmm.

160. 29:38

     It's, it's bringing you to the point, it's cool. But, uh, what Cursor does is to basically give you a plan. That's it.

161. 29:46

     Just an execution plan.

162. 29:47

     Right.

163. 29:47

     Okay.

164. 29:48

     Yeah.

165. 29:48

     Oh, I see. So I think that the fundamental difference there, uh... Does that plan get committed anywhere, or is it just ephemeral?

166. 29:57

     Uh, it's kind of [inaudible].

167. 29:59

     Okay. So what I want over time is not, is not just how we make the changes we care about, but it is actually the documentation and specification about what the system does.

168. 30:09

     Um, so the long-term goal I have is that as Kiro, we are able to do sort of a bidirectional sync. That is, as you continue to work with Kiro, you're not just accruing these sort of task lists.

169. 30:21

     Uh, and so I'm just gonna say go for it to go to the tasks. Um, but we're not just accruing task lists, but actually, if I come back and, let's say, change the requirements down the road, we will mutate a previous spec.

170. 30:32

     So I'm looking at really just a diff of requirements, which, as you go through the greenfield process, you're gonna produce a lot of green in your PRs, which is maybe not the best, 'cause then I'm just reviewing [REDACTED:generic_id] new huge markdown files.

171. 30:43

     But on the next time or the subsequent times that I go and open that doc up, I want to be seeing, "Oh, you've actually r- you know, you've relaxed this previous requirement.

172. 30:53

     You've added a requirement, and that actually has this implication on the design doc." Um, that is the process the Kiro team internally uses to talk about changes to the Kiro system.

173. 31:01

     So we review... Our design docs have, in general, been Uh, replaced by spec reviews. So we will, you know, somebody will take a, a spec from markdown, they'll blast it into our wiki basically using an MCP tool we use internally, and then we'll review that thing and comment on it in a, in sort of a design session

174. 31:20

     as opposed to, you know, this markdown file or a wiki from scratch. Um, so it becomes sort of if, uh, well, it's actually not like an ADR because it's not point in time.

175. 31:30

     It is like this living documentation about the system. Um, but yeah. Yeah. Thanks for the question. There's one over here.

176. 31:39

     Um, this may be more a spectrum development question, but are there like, like, is there like a template for a set of files that you fill out? Like right now you're in the design.md.

177. 31:51

     Mm-hmm.

178. 31:52

     Are there like... Is this, is the design.md the spec and it's a single doc or are there like-

179. 31:59

     Oh, great question. So the-- Yeah, the question was, um, are there, uh, and correct me if I'm wrong here, but question is, are there a set of templates that are used for the system?

180. 32:08

     And is the question you're driving at, can you change the templates or is just are there? Okay. So the, yeah, question is, are there a set of templates? Um, there are implicitly in our system prompts for how we take care with your specs.

181. 32:20

     So you'll see here at the top nav bar here, right now we're really rigid about this requirement design task list phase, but we know that doesn't work for everybody.

182. 32:28

     For example, if you're starting-- We get this feedback from a lot of internal Amazonians actually, that I wanna start with a... I have an idea for a technical design, and I don't necessarily know what the requirements are yet, but I know I wanna make...

183. 32:39

     Maybe design is even the wrong word. I wanna start with a technical note. Like I wanna refac- This comes up a lot for refactoring actually. Um, so I wanna refactor this to no longer have a dependency on, um, here's a good example.

184. 32:54

     Here we use a ton of mutexes around the system to make sure that we're locking appropriately when the agent is taking certain actions because we don't want different agents to step on each other's toes.

185. 33:02

     But maybe I wanna challenge the requirements of the system so I can remove one of these mutexes, uh, or semaphores I should say. Um, so I might start with something like a technical note, and then from there sort of extract the re- the requirements that I wanna share with the team and say, "Hey, you know, I had

186. 33:17

     to kind of play with it for a little while to understand what I wanted to build, but I still wanna generate all these rich artifacts." So today it's this structured workflow.

187. 33:25

     We're playing a lot around with making that a little bit more flexible. But the, the structure is important because the structure lets us build reproducible tooling that is not just an L- So I think that that's an important distinction we make, is that our agent is not just an LLM with a workflow on top of it.

188. 33:40

     The back end may or may not be an LLM, or it may or may not be other neurosymbolic reasoning tools under the hood. Um, and so we, we try to keep that distinction a little bit clear, uh, that you're not just talking to like Sonnet or Gemini or whatever.

189. 33:54

     You're talking to sort of an amalgam of systems based on what type of task you're executing at any point in time. Um, although when you're chatting, you are talking to just an LLM. [laughs]

190. 34:04

     Um, but yeah, so we have a template for the requirements. We have a template for this design doc because there's sections that we think are important to cover. Um, and again, like if you disagree and you're like, "I don't care about the testing strategy section," just ask the agent to do it.

191. 34:19

     And similarly, the task list has, is structured because we have sort of UI elements that are built on top of it as well, is like task management and, um...

192. 34:27

     Do we have-- We'll get there when we do some property-based testing, but, um, there's some additional UI we'll add for things like optional. You can have optional tasks and stuff like that.

193. 34:38

     And so we, we need the structure there for our, uh, task list LSP to work, for example. Um, yeah, thank you for the question. Anything else before we truck on?

194. 34:50

     Cool. Uh, I may need somebody to remind me what we were doing. Oh, that's right. So we went through and we synthesized the spec for adding memory and some amount of persistence to my agent.

195. 35:02

     By the way, I didn't introduce you to this project. This project is called Gramps. It is, uh, it is an agent that I'm deploying to AgentCore to learn about it.

196. 35:10

     I mentioned that, but what I didn't tell you is that is, it is, uh, a dad joke generator. A very expensive one since we're powering it via LLMs. But effectively, you're a dad joke generat-generator.

197. 35:25

     Jokes should be clean. They should be based on puns, you know. Obviously, bon-bonus points if they're slightly corny but endearing, um, yada, yada. So we're deploying this to the back end.

198. 35:34

     So the reason I want memory is because every time I ask the dad joke generator for a joke, it gives me the same damn joke, and that's just super boring, and my kids are not gonna be excited about that.

199. 35:43

     So I want memory so that as I come back for the same session, I get different jokes over and over again. Um, that's the context on the project. So we've come through here and we actually said we generated this thing.

200. 35:54

     We did the task list. I said, "Hey, is this the idiomatic way to do it?" But what I know is that we didn't actually, uh, we're not using Agent Core's memory feature, which is probably a big oops.

201. 36:05

     Um, and so, you know, quick show of hands, do we wanna make the mistake and go all the way to synthesis and deployment, or should we fix it now?

202. 36:11

     Who wants to fix it now 'cause we know better? No, I wanna make the mistake. Let's keep on trucking. I, I had [REDACTED:generic_id] yeses in a room full of nothing.

203. 36:19

     So we're gonna make the mistake and then come back and fix it later. So, uh, let's say run

204. 36:26

     all tasks in order. Uh, the reason I mention in order, which seems very specific, is because this is a preview build of Kiro. Um, and so somebody just added to the system prompt, "I should only do one task at a time," and I found that if I say run all tasks, it thinks I somehow mean do them

205. 36:45

     all in parallel. So we'll-- that'll be fixed before these changes get out to production. So Kiro's gonna keep kinda going through here and chewing away on the system in the back end.

206. 36:54

     Um, it has steering docs that explain how to do its job. It has, which I guess I should show you guys, steering again is like memory. So I have some steering on how to do commits, uh, you know, how I like to have commits, but also steering on things like how do you actually deploy this thing?

207. 37:10

     Um, how do you deal with AgentCore? And then how do you run the commands that are necessary for you to deploy this to my local dev account? Um, and th-those are mostly just an example again of sharpening your tools.

208. 37:21

     Like, uh, I went through this kinda painful process of figuring out Oh, you know, you have to use this parameter on the CDK, y- the CDK command. You have to use this flag, otherwise it doesn't work correctly.

209. 37:32

     And so once I go through that pain of learning, I just say, "Kiro, write what you learned into a steering doc," and it will usually do a very good job of summarizing.

210. 37:39

     Um, and so it generated automatically this AgentCore LangGraph workflow MD file. Um, yeah. So I mean, it's just gonna kinda go away here and truck, truck on and do its job, and we can watch it in the background.

211. 37:52

     But in the interim, um, I think at this point we're at a pretty flexible spot. Uh, so for folks who want, feel free to use Kiro, try out specter and dev on your own.

212. 38:00

     I'm gonna keep just kind of running this in the background and taking questions and comments, but that's kind of it for the scheduled part of today.

213. 38:09

     Yep.

214. 38:10

     How does Kiro work for, like, existing large code bases or this-

215. 38:14

     Yeah.

216. 38:15

     That.

217. 38:16

     Yeah. Question was, how does Kiro work for large and existing code bases, basically the brownfield use case. Uh, and the answer is, it depends on what you're trying to do.

218. 38:23

     Um, for spec-driven dev, you can ask Kiro to do research into what already exists. So when you start a new spec, it will usually start by reading through the f-- the working tree.

219. 38:31

     Um, but the agent is generally starting from a, a scratch perspective, right? It needs to understand the system. Um, in practice, what that means is that you're going to end up with a bunch of things like if your system already had good separation of concerns, uh, your-- the components in your system are highly cohesive, and they're sort

220. 38:49

     of-- or highly coherent and highly cohesive, it's gonna have a great job, right? It's gonna be able to say, "This is the module that does this thing. I don't need to keep eighteen things in my context to do my job," and it's gonna do well.

221. 39:01

     Um, if you, uh, let's just take a, an example that's off the top of my head. If you were trying to launch an IDE very quickly, uh, leading up to an AWS launch, and you, um, you know, took a lot of tech debt along the way that you need to unwind, and, you know, nobody here would do

222. 39:17

     that, I'm sure. But, um, in case you did that like me, then your agent might actually have a much harder time traversing the code base in the same way that a dev would, right?

223. 39:26

     So, uh, from just kind of that perspective, the more reliable things like your test suite are and the more understandable things like module separation and sort of decomposition of concerns are, the better the agent will do.

224. 39:38

     Um, and reverse is true, of course. Now, for things like, uh, understanding the code base, this is a bad example because this is a very small code base. But, uh, we do have things like, you know, code search and workspace, um, uh, I don't know what to call these, context providers.

225. 39:59

     Um, so you can come in here and just say, "I wanna do code." Oh, what is it?

226. 40:06

     I might have turned this off, actually. Oh, I did turn it off because the code base isn't big enough. We'll do things like indexing in the background so the agent, like, you can do semantic search over what you've got, um, if you're just chatting.

227. 40:18

     But in general, uh, Kiro should go in and do sort of background search to figure out how to do its job. Like, as the code base scales up, it's gonna be less-- do probably less well overall, but that's one thing we're working on as a team.

228. 40:31

     Yeah. Did that answer your question, or did I kinda glance off the side a bit?

229. 40:36

     Yeah, I think I got it.

230. 40:36

     Okay, cool. Anybody else?

231. 40:40

     With the, uh, uh, repo size.

232. 40:44

     Uh, how long are you willing to wait for indexing to complete? [laughs]

233. 40:49

     Uh, so one example I have is that the CodeOSS, um, if it's not supremely obvious by looking at it, Kiro is a CodeOSS fork, just like, you know, Cursor and Surf.

234. 40:59

     Um, one of the challenges we've had is the CodeOSS code base is very large, fairly large. There's other big ones out there, but that's kind of my large code base because I'm not forced to get to work in it fairly frequently.

235. 41:13

     Um, and so there, there's definitely some perceived slowdown when you're dealing with something large like that, especially when you talk about code base indexing. It's a very active area of work for us, though.

236. 41:23

     So we're trying to do things like, um, either remove indexing from the critical path so that you're not waiting there on some kind of slowed down render thread because indexing is running.

237. 41:34

     Um, but in practice, there should not be. I mean, again, the agent may practically do less well, but we're gonna be talking in a couple weeks at re:Invent about how some of the temporal features in Kiro were built via spec in a code base we did not understand particularly well because we're just not VS Code devs.

238. 41:51

     Um, and Kiro did a fine job of it. But again, that's a testament to the fact that code base is reasonably well, um, structured. And, like, if you've taken the time to understand how it works, it's very understandable.

239. 42:04

     If you've not, it will-- might be a little bit opaque to, to stare at.

240. 42:09

     Yep.

241. 42:10

     Uh, in terms of indexing, is this like just, just putting, um, uh, as, as, as much information from the code base into context, or it just-

242. 42:19

     Mm-hmm.

243. 42:19

     Is there a way to, like, create some kind of, like, vector database of all the, uh, uh, code base and then, like, query it? Or just-

244. 42:30

     Yes. Um, so the question was, what do you mean by indexing? Um, 'cause indexing can mean a bunch of different things. And what I mean is that, um, the agent is actually not provided the index.

245. 42:42

     We wanna keep the agent context as small as possible. We use the, uh, the index for most, like, secondary effects. Things like if you're doing a, uh, a code search or if I do something like search for, um, pound, uh, what's a file in here?

246. 42:58

     Issue server. Like, we use it more for these types of UI, um, than giving it to the agent because the agent does... This is sort of anecdotal and based on our benchmarks, does better when given less context, but given the tools to understand where to go find things.

247. 43:13

     Um, something we've heard a lot about is sort of incremental disclosure here at this conference, and that's, again, we don't wanna load too much at the beginning of the context and conversation with the agent.

248. 43:21

     We want the agent to self-discover the right context for the task. Yeah.

249. 43:26

     Thank you. Yep.

250. 43:29

     You guys managing session length? Like, is there any kind of compression or pruning or offloading? The regular cab [audio glitching]

251. 43:36

     Yeah. So, um, the question was how do we manage session length? We have no incremental pruning today or incremental summary. Um, you basically just accrete context until you hit your limit, which I think right now I'm on auto, which has like a two hundred K token limit, um, similar to the sonnets.

252. 43:54

     Um, uh, so we don't have a very sophisticated algorithm here yet. We've looked at a few things, but our number one concern actually is, um, prompt caching hit rate.

253. 44:04

     And so in a normal use case, I can achieve something like ninety, ninety-five percent cache token usage here on per turn, which means that my interactions are very fast.

254. 44:13

     And that's-- or they're much faster than the alternative, which is I'm sending one hundred and sixty K tokens to b- to Bedrock cold. Um, so that's one of the reasons we've actually not done much experimentation with incremental summary.

255. 44:25

     Um, our summarization feature exists when you hit the cap. It's not great. It's something we're trying to, uh, ship an improved version very, very shortly, um, e.g., in the next couple of weeks, which should be faster.

256. 44:38

     Today, it's like a one-off operation that can take up to thirty or forty-five seconds, which is a horrendous experience. We're hoping to fix that here and make it sort of a real-time experience.

257. 44:48

     Just as a follow-on to that-

258. 44:49

     Yep

259. 44:50

     ... managing statefulness between sessions then, is that how you-- while you're relying on this area of being made it spec-driven [audio glitching]

260. 45:01

     So sort of. Um, that is not the only reason. I mean, the spec-- the spec-driven dev is less to do with performance and more to do with reproducibility and accuracy of the agent.

261. 45:13

     Um, because if we can give you the right result, the, the, the way I, and I think that we talk about it internally as this team is, if I spend ten seconds giving a prompt to the agent and then it goes off and it gets it wrong, it's like, eh, it's kinda no skin off my back, right?

262. 45:29

     I burned however many tokens and, you know, a couple cents of credit usage with whoever my LLM provider is, but I spent ten seconds generating a prompt. If I spend five to ten minutes with the system producing a detailed design doc, or let's just say even a detailed set of requirements, I want it to do a fairly

263. 45:47

     good job. If I spend an hour generating a design doc, reviewing it with my team, and then synthesizing from that, I want it to get it right. So the goal necessarily is not just latency, but actually accuracy when we talk about that.

264. 46:00

     Yeah.

265. 46:00

     So it's a both/and. You need to do both. But, um, spec comes more from a con-- uh, the goal to have, um, highly reproducible output.

266. 46:12

     I'm gonna go over here first and then you.

267. 46:14

     Yeah. How do each of these task agents pass context to each other? And then are you only supposed to run this, this parent task? Because it just finished all like [REDACTED:generic_id] point one, [REDACTED:generic_id] point two, [REDACTED:generic_id] point [REDACTED:generic_id], but then it still thought that [REDACTED:generic_id] point one wasn't done and ran that and [REDACTED:generic_id] point two.

268. 46:30

     Oh, did it?

269. 46:31

     Yeah. Well, no, mine right-

270. 46:32

     Oh, okay. Yeah, yeah. Um, so if you--

271. 46:39

     The, uh, the question is if you're in the UI and you're like running tasks, and I can just kinda pull up my task list here. Um, so if I just hit start, start, start, each of these is going to be a new session, which means the context is completely unique.

272. 46:52

     Um, personally, I like to just... if I can-- if I've got the context base to afford it, I just say do all the tasks 'cause I find that more understandable, and I think I actually get better performance.

273. 47:01

     But by default, each task will be a new session that has no shared context with the previous ones. So the session is effectively just seeded with your specification and then, like, here you're working on a spec that does all this stuff, block of text, um, and you are doing this task, da, da, da.

274. 47:18

     Don't do any other tasks. Just do this. Um, so that sounds like a bug. Um-

275. 47:22

     There has been a sub-agents for, for certain things.

276. 47:25

     We don't have sub-agents yet in Kiro. Some we're working on. Yeah. Yeah, 'cause I mean, ideally, right, if we click on task [REDACTED:generic_id], and I've got [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] and they're w- separated, there's no good reason they couldn't have different systems working on them.

277. 47:38

     Yeah. Uh, right here.

278. 47:42

     We do have in the Kiro CLI custom agents that you can also run off.

279. 47:47

     Yeah. Kiro CLI is a concept of custom agents, um, which can be run sort of as a task. Um, and it's something we're playing with right now in Kiro Desktop.

280. 47:55

     Um, and I think you have another one.

281. 47:57

     Yeah. I'm sorry if I missed this, but in the spec folder, um, as you do more and more of these tasks over time-

282. 48:05

     Yep

283. 48:06

     ... is it, is it just all in one design requirements tasks, your whole project is defined there, or is it, is it grouped by like-

284. 48:13

     Oh, that's a good question. Um, yeah, so I will have many... I will have, uh... The question was, as you do more, you generate, let's say, more specs over time, are you sort of just creating one massive spec?

285. 48:25

     And no. Uh, let me open a different project.

286. 48:48

     Um, so this is, for example, the Kiro extension, which is like a one B extension inside the Kiro IDE. This is where the agent itself lives. And so we have pruned some specs, but there are specs in here that we can talk through or I can just kinda demo.

287. 49:02

     Um, so these are-- The way I think about it is the spec sort of represents a feature or a problem area in the, in the project. And so, for example, I can blast this a little, uh, larger.

288. 49:14

     So for example, we have, um, like some of these are just tests. We've done things like, oh, could we have a prompt registry? Could we have a prompt registry file loader?

289. 49:23

     They may or may not make it all the way to production. Um, I want telemetry on the chat UI. So these are just like, somebody will go off and spend, or maybe represents a few days of work for an SDE.

290. 49:34

     Um, Agents MD support is a good one where we just, you know, I sort of said, "Research what Agents MD is and build it in the way you build steering in, like support in the same way."

291. 49:43

     The spec is fairly unlikely for us to come back and revisit in the future, so I may actually just delete it, um, which is what we've done with some of the older ones.

292. 49:49

     But a good example of one that we might come back to is our message history sanitizer. So one thing we've had issues with or we had issues with early in the vel- the development of Kiro is that we would send these sort of invalid, um, sequences of messages because, let's say, the Anthropic API required tools to be

293. 50:07

     in the same order they were invoked in the responses, but the system wasn't doing that. So we built this whole sanitizer system that has a bunch of requirements around, um...

294. 50:17

     Let's see. Very specifically... Yeah. When conversation is validated, the system shall verify that each user input is either non-empty content or tool responses. So we had things where like empty strings would get passed in, but there was a tool response.

295. 50:31

     This is a good example where we've come in over time and actually just added, maybe not to the requirements, but to the, to the acceptance criteria of the requirements as new validation rules are uncovered.

296. 50:42

     Yeah.

297. 50:42

     So how do you handle, like, that? So for example, you have like-

298. 50:47

     Yep

299. 50:47

     ... telemetry up there.

300. 50:48

     Yep.

301. 50:49

     You add a feature that needs telemetry. Is it gonna go back and update that spec too or you're just-

302. 50:53

     It should, yeah. So if you... Usually you'll see... And let me just ask, uh,

303. 50:58

     a new chat here. Um. No, that's a terrible idea.

304. 51:13

     So let me... Back. So here I've asked, I've made a, in spec mode, I've made some requests to, um, add UI telemetry to the thing. I'll help you add it.

305. 51:28

     Let me first check if there's any relevant runbooks, then explore the code base and send the implementation. It might go do a little bit of research here, da, da, da.

306. 51:36

     And then flip of a coin, again, it's an LM, so it may or may not discover the existing te- uh, spec. But ideally, it will, after doing its research, say there exists a spec already for things like UI telemetry.

307. 51:49

     I'm gonna go and amend that one. Um, and if it doesn't in this case, like I would come in and just ask it to, um, as sort of the operator for the system.

308. 51:57

     But over time, again, we want that to be easier for you as a user to not have to think about so much.

309. 52:04

     We can watch it while it chugs along.

310. 52:10

     Is there anything reconfigured in Kiro that makes it better to work with AWS? So trying to, you know.

311. 52:18

     No, not really. Um...

312. 52:22

     Was that a question?

313. 52:23

     Oh, uh, question was, uh, is there anything in Kiro that is preconfigured to make it work better with AWS? No. Um, we are, uh, sort of purposefully, we're in, we are brought to you by AWS, which so, you know, uh, Andy Jassy and Jeffy B pay my check.

314. 52:39

     But, um, we're not like an AWS product that's deeply, deeply integrated with the rest of the AWS ecosystem. Now, that said, I still answer emails when somebody says, "Why is this other thing we built with AWS not working with Kiro?"

315. 52:51

     Yay. But, um, similarly, like if you're building on GC or Azure, whatever, um, or you're running some on-prem system, the product should work just as well for you. That's our goal.

316. 53:02

     A good, a good answer potentially is the AWS documentation MCP server.

317. 53:07

     Yes.

318. 53:08

     So there are MCP servers that you can add into any of these specs that will make them better at working with AWS stuff.

319. 53:14

     Yeah, that's a good point. So like in this case, I actually had to add the AWS MCP documentation here. We could, of course, have natively bundled this, but I don't wanna ship this to customers who don't need it.

320. 53:27

     Yeah. 'Cause again, AWS is not the only docs that we might care about. Um, by the way, coming back to your question, so it did find the existing spec for telemetry.

321. 53:35

     It read it, it read different sections of it, and now it's actually making amendments to it, so we can follow the diff as it shows up here. So it's added new, uh, requirements, um, to the preexisting spec.

322. 53:45

     So this is effectively another case where we're mutating the system as opposed to just adding this sort of never-ending spiel of specs.

323. 53:54

     I guess what I'm, I'm wondering is like how does it know or decide where to put the spec? You know, if you break down your project into these different categories-

324. 54:04

     Yep

325. 54:05

     ... I would imagine like crossover.

326. 54:07

     Yeah, I mean, it's, that, that's sort of like software development in a nutshell though, right? Like how do you actually define the seams between different parts of your system, different concerns, the product.

327. 54:17

     Right. But if you want to like build something, like I have a task and it's gonna cost, require changing like [REDACTED:generic_id] or four things.

328. 54:23

     Yep.

329. 54:23

     You're gonna change [REDACTED:generic_id] or four specs and then run tasks across [REDACTED:generic_id] or four specs?

330. 54:27

     Oh, yeah, yeah. No, it should not do that. It would probably... So again, I don't have a good example offhand that we can do for that. But, um, my, my perspective would be that if you're working on something that is cross-functional...

331. 54:38

     Uh, by the way, the question was, um, if I'm working on something that, let's say I have a spec for security requirements and I have a spec for API design, uh, like the API shapes, and I have a spec for logging, and I am changing something in the API public interface that is a security-facing concern because we're

332. 55:01

     redacting logging PII. Um, I think that's maybe a semi-tangible use case, uh, that we can all imagine coming down from our governance teams. Um, I want to...

333. 55:13

     I would imagine that you either pick one of those to load the requirements into, or you create sort of a cross-functional spec. But that would come down to, I think, you as a, as an operator making that decision.

334. 55:23

     In much the same way that if I-- how you actually implement it might be you, you would not necessarily implement my PII, API redaction module as a standalone thing.

335. 55:34

     It's gonna be a cross-cutting theme across your code base, I'd imagine.

336. 55:38

     This is also a good example. There's like multi-root workspace came out when it went GA on Monday, and now you can like drive different... So like in your example, you just went through with like APIs and Auth and like even the front-ending.

337. 55:50

     You can bring in those projects if you have them separately and then still work [audio cuts out]

338. 55:57

     .

339. 55:58

     Yeah. Thanks, Brooke.

340. 56:04

     Uh, the mental model. If the spec generates the code after that, like what flow you can specify? How does that work?

341. 56:12

     Yeah. So, um, we have now synthesized effectively the spec. So we, we sat down, we defined the requirements, design, and task list. I've had Kiro now go through and run all the tasks in this spec.

342. 56:25

     So it ran them one at a time. It basically worked on small bite-sized pieces of work, uh, chunk by chunk. And then, uh, now this is done. So what we've actually produced is not just like the completed spec, but it went here into my agent, and it did a few things.

343. 56:42

     In the CDK repo, because it's doing persistence to S3, I'm sure it added a bucket. Yep, some new bucket encryption and yada, yada. It then went in to the agent, added the S3 checkpoint saver.

344. 56:53

     It looks like it, you know, created a check pointer. It adds this to the graph, and it kind of passes this all the way through the system. And the S3 check pointer here, I'm sure, has some knowledge of how to write the checkpoints to and from S3.

345. 57:07

     So like we have gone not just for defining the system, but we've now, um, produced it end to end, or we've, uh, delivered it end to end, including property tests, I believe.

346. 57:16

     Um, yeah.

347. 57:20

     Oh, I have a answer to an earlier question related to like, um, some specific AWS related features like that makes it easier to work with. The Kiro CLI comes with a use AWS tool which helps with the AWS CLI.

348. 57:36

     Yep, yep. So, uh, what Rob's pointing out is the Kiro CLI, which we just rebranded, um, this week, has a use AWS tool, which is basically a wrapper over the AWS SDK, um, to make some of those things easy.

349. 57:47

     Uh, but again, BYO use GCP tool as an FCP server if you were so inclined, if that's your, uh, tool of choice. And I believe, don't quote me on this, um, 'cause the CLI is kind of new, new to my-- new to me, I should say.

350. 58:03

     Um, but I believe you can turn off tools in the CLI as well. Let me know if that's not right, Rob.

351. 58:08

     Yeah, you can turn it off like that.

352. 58:09

     Yeah. So that's like you're actually not strict. Uh, in the desktop product today, you can't control the tools, the native tools built in, but in CLI you can.

353. 58:21

     Um, so I, I intuitively get the benefits of having a spec. Have you done any work to empirically see like how a project or a problem would work with or without a spec?

354. 58:34

     Yeah. Um, we do have benchmarks, uh, covering the data offhand. Um, I think part of that's in our blogs. So if you go to the Kiro.dev/blog, or it's on the site, we, we talk really crisply about some of the lift that things like property-based testing give to task accuracy.

355. 58:51

     Giant's team's always working on that stuff.

356. 58:53

     Uh, I remember a blog about specs when Kiro was first introduced two decades ago. Um, [audio cuts out]

357. 59:00

     .

358. 59:00

     Yep. Distinguished engineer for databases. Yep.

359. 59:03

     His, like, blog post really sums it up, and I think it has the data specifically that you are asking for, but I think it will be useful.

360. 59:10

     Yep. Yeah. Yep.

361. 59:13

     How does it work? I, I understand the feature side of it, but how does it work in a non-functional side, like latency,

362. 59:21

     dealing with, you know, a little bit more harder problems?

363. 59:25

     Well, yeah, I mean, that is ultimately the goal here, right? Is we're saying you're making a slightly larger investment up front, but we believe that the, uh, the structure we're bringing is going to help you get-- increase the accuracy of your, uh, result.

364. 59:37

     So, um, while we've got a team of people who are basically working on making spec better, my job, when I fly back to Seattle, is to make Kiro as a whole much faster.

365. 59:45

     Um, one, execution time and like, kind of like lagginess in the UI. But two, how do we get tokens through the system faster? How do we get responses to you faster so that, like, you're not sinking as much cost into Kiro to use a spec?

366. 59:58

     Yeah, yeah. Fair. I'm not talking about the Kiro tool itself.

367. 1:00:01

     Okay.

368. 1:00:02

     Code generated from the spec.

369. 1:00:04

     Oh, oh, okay. Yeah, you mean like the non-functional requirements of the generated code. So, uh, that's gonna come down to, I think, what you're specifically trying to do. So you could add, uh, one of the slides I had here was talking a little bit about how to tweak the process and tweak the artifacts for your use cases.

370. 1:00:21

     Um, again, you could very easily add something like, "I want non-functional requirements for speed and runtime and things like lock contention to be considered in the design phase." Um, yeah, that's something you could certainly add.

371. 1:00:33

     So you could generate a code in Rust or, uh, or Java?

372. 1:00:37

     Yeah, totally. Yeah.

373. 1:00:39

     And it will vary in the functional depending on what language it generated.

374. 1:00:45

     I mean, it would, it would have to. Like, yeah, there's no other way, I think, to approach it. Um, again, I'm just-- I'm familiar with Node, so I'm doing everything here in Node, but you can use this with any language.

375. 1:00:53

     I think technically we say we support Java, Python, JavaScript, um, and type-- Jesus. JavaScript, TypeScript, Java, and Rust. But in practice, there's no reason that this doesn't work with any language.

376. 1:01:07

     I mean, it's just an LLM. The g-- there's nothing language specific or framework specific in the system. And for those of you, um, so there was a conference earlier this week hosted by Tessell, which are doing sort of specs for knowledge base.

377. 1:01:19

     Um, as long as you've got the right grounding docs in there, and this is sort of the, uh, their argument is that it should not matter what you're building.

378. 1:01:27

     Like, that's all just informed by the, the context you're building for your system.

379. 1:01:32

     This is also a really good point for steering. So steering, you can get the agent to develop code in the way you want. Like, being a developer is all about making trade-offs, and the problem with Kiro out of the box is it's like so polite 'cause it's trying to be everything to everyone.

380. 1:01:46

     Um, and especially like with latency and cost and other things like that, just tell it in steering What you want it to prioritize, and then that will influence any code that gets generated.

381. 1:01:56

     Yep.

382. 1:01:56

     Even, like, how it designs based on that as well. So if there's something that's very specific to your use case or your industry or whatever, just shove it in that steering file and then it will be more engaged with-

383. 1:02:05

     Yeah, that's exactly right. So for example, I, I will have Kiro generate, um,

384. 1:02:11

     commits for me, and one of the things I care, I personally care about is that I can track commits I generate versus commits that Kiro generates, being the ones that come from the system.

385. 1:02:20

     And so my steering doc, while short, includes things like very specifically my requirement for Kiro is... Let's just use the UI.

386. 1:02:32

     Um, da, da, da. Attributed to the co-author of Kiro Agent, um, which is trivial, but also I want it to happen every time. So in this case, it just generated a commit co-authored by Kiro Agent, da, da, da.

387. 1:02:45

     So that's an example of, like, you could add whatever you want in there, not just something related to git commits. But you could do code style, you could do, um, uh, you know, code style, code coverage.

388. 1:02:56

     Uh, whenever you add a spec or you're adding a new module, make sure that you annotate it with coverage minimums that are 90%, because that's the thing I care about.

389. 1:03:03

     Um, you can kinda put anything you want up in there. The good news is it looks like what we built works. Um, Kiro's very happy with itself at least, and it looks like all tests passed.

390. 1:03:14

     But, um, yeah, so we'll... We can deploy this to the back end and see how things work.

391. 1:03:21

     We're, uh, technically just about at time, so, you know, if anybody has any other questions, I'm gonna stick around here for a while. But, uh, thank you all for joining, listening, and, uh, learning a little bit more about Spectre and Dev. [upbeat music]