Amazon AGI

Talk transcript

1. 0:00

   [upbeat music] Today, I'm gonna be talking about agents, codebases, and teams.

2. 0:16

   Essentially, how do you get your team to actually ship together with agents? And, like, I think for the longest time, like, the one thing that's bugged me is, like, there's so much content about, you know, how do you set up your own codebase to, like, work well with agents?

3. 0:29

   You know, like, what skills do you add? You know, this skill's better, that setup's better. But it all seems to break the moment you, like, actually try to use it with your team in your actual production setup.

4. 0:39

   And, like, for individual repos, it kind of makes sense. But the moment you actually try to use it with your own, like, team setup, it tends to break. And, like, I think over the past few months, I've figured out how to make it work, um, with a team of, uh, folks.

5. 0:53

   So like, I was leading a team of, like, ten people over the last few months, and I think we found, like, a good solution, and I want to share that with you guys.

6. 1:00

   But before we get into that, I just want to recap, like, you know, what's been the journey that we've been on, right? So we started, you know, code-coding agents took off, and a few people got really, really good leverage, right?

7. 1:12

   And I think all of us were kind of asking, like, "Is this AGI?" You know? Like, did we achieve it? Then, you know, companies took that and said, "Well, if one person can do so well, let's just get everyone and let's mandate it and, like, token max," right?

8. 1:27

   And that was clearly a, a ga- a galaxy brain moment, right? And then the inevitable happened, you know, AI slop shipped, and there was a bunch of SEV 2s.

9. 1:36

   I'm not [chuckles] gonna name which companies. But, you know, essentially you saw people kind of retracting. They said, "I don't think, [chuckles] I don't think this is the best option here."

10. 1:45

    And eventually, model prices cl-- you know, climbed. We saw, you know, people figured out that tokens have to be paid for. Like, you just can't token max your way through life.

11. 1:55

    And budgets got bolted on, and, you know, essentially money is being lit on fire, and the money has to come from somewhere. So given this journey, I wanna actually, like...

12. 2:04

    This is the enterprise journey, right? And what does that do for a single developer? And I think this is an important framing 'cause it really talks about people as a part of a team, right?

13. 2:15

    And I wanna look at it from, like, two axes. So there is the fear axis, where people li-lie on the spectrum, right? They're coming-- It's like, is it coming from my job?

14. 2:24

    Like, am I gonna be out of a job? Or it's, like, a really handy tool, and they're really-- you know, they're not that fearful. Versus, like, the confidence they have in how much they're executing it.

15. 2:32

    So they can either use it a lot or they can use it, you know, not that much because they don't really know how to use it that well. Now, when we started, like, people said, "Oh, you know, what is this?

16. 2:43

    Is this the end?" Like, "Am I needed?" And fear was pretty high, utilization was pretty low 'cause people didn't really know how to use it. And then when a few people got outsized leverage, you saw, you know, early adopters.

17. 2:55

    Basically, people saw them, and people said, "Okay, well, it looks like I'm still kind of needed if I figure out how to use this thing, so let me actually try using it," right?

18. 3:04

    And then we saw, you know, mandates and token maxing, and people kind of got a little skeptical. Like, confidence stayed the same, but people tried to use it a lot more, right?

19. 3:14

    And then we realized, ah, you know, there's a bunch of slop shipping, there are SEV 2s, and

20. 3:19

    I-- you know, it's like, I'm not really that scared 'cause, you know, it just slip-- it just ships slop. Like, I'm gonna still be needed. And they don't even know how to use it that well 'cause, like, now the confidence is cratered, right?

21. 3:29

    And so you gotta figure out how to get people from all the way, wherever they are on the spectrum, to where they're not fearful, and they're actually using it a whole lot more.

22. 3:39

    And this is the framing that I wanna, like, I want everyone to keep in mind as they're actually trying to get a team to adopt good, like, AI usage and good AI patterns, right?

23. 3:49

    And so the question is, you know, what does it take? Step one, create a Claude MD. Step two, add some skills. Is that it? Did we, like, solve it?

24. 4:00

    You know, [chuckles] I think we all know. Like, you guys are here because, like, clearly, life's not that simple, and, like, stuff's messy, right? And I think a few people might ask, you know, why doesn't this work?

25. 4:11

    Like, isn't that what, like, everyone does? And I wanna, like, just talk about a few things you might see that actually indicate that, yeah, like, this isn't working.

26. 4:22

    So the first thing is, if you're babysitting your agents, it's not the right setup, right? And you gotta realize that. If you're seeing people in your team babysitting their agents, something's wrong.

27. 4:34

    Is-- You know, one of the things that I heard a lot was, is, you know, insert whatever latest model there is being really dumb today. The model didn't change, right?

28. 4:42

    The hardness may have changed underneath. But if it's really, like, that's, that's susceptible to, like, small changes in the hardness, clearly your own codebase isn't set up well.

29. 4:54

    It's silently burning context and money. Like, you don't realize it. You know, you go-- You blow through, like, 500K context. You might go to, like, 750K, a million and hit auto compact even though you're not doing, like, a really complicated task.

30. 5:06

    Clearly, something's wrong. You know, if you have long-ass sessions needing constant intervention, there's still something that's wrong.

31. 5:15

    If you're getting a constant slop factory, [chuckles] you obviously know, like, things are, things are not good. And if you're asking-- If you find yourself asking, you know, how are these other companies shipping so fast?

32. 5:25

    Like, how are model companies releasing models at, like, a month-and-a-half, two-month cadence? Clearly they, they have something which we don't, right?

33. 5:34

    And so I guess everyone's kind of thinking, "So, you know, how do we solve this correctly?"

34. 5:40

    And so I think the first thing to realize is we need to frame it correctly, right? It isn't really an IC's job. It's a job for leadership. It's a job for the company, right?

35. 5:51

    Making engineers work well with their agents is truly at-- the most impactful thing you could do as an organization because that's gonna enable, you know, your engineers to ship faster and with confidence and avoid, you know, a lot of incidents.

36. 6:06

    If we live in this f- figure-it-out-for-yourself paradigm, like, people are gonna get outsized productivity, some people aren't, and, like, the people who are generating, like, 10 PRs a day are gonna, like, look like, you know, gods compared to people who are shipping, like, one to two.

37. 6:20

    And the one-to-two PR people are actually gonna get left with the review burden, and that's actually a really, really bad thing. 'Cause now not only can they not ship, they're gonna actually see bad code and then curse the agents and hence not be able to get onto the, "Let's ship 10 PRs," right?

38. 6:36

    And so it's really important to do this. If it's a problem facing the team, there's a few things you can do, right? The most impactful things that you can do to set up your code base to, like, make it work well require team buy-in.

39. 6:49

    You can't just, like... If you wanna change the way your code base is organized, you can't do that as an IC, right? And if you treat it as a leadership problem, then you can do things like this.

40. 7:01

    So the other thing this needs is Harness engineering, right? Per code base. And I think there's a lot of content on this, so I just wanna talk about a few principles.

41. 7:10

    But I don't wanna, like, make this talk about that because there's a lot of smart people. You're at AI Engineer. Like, this, this conference is all about people telling you how to u- best set up, you know, your code base to make things function well.

42. 7:21

    So I don't wanna talk too much about this, but there's a few key principles here. Smart prompt injection is one of them. You wanna treat your entire code base as one way to that so that you're able to, like, smartly prompt inject the model with just the right context at just the right time without you needing to

43. 7:37

    do it, and that's the framing. You wanna be able to say, "Okay, I've set it off on this task." It has, like, a map of how to find the things it needs at the time it needs it.

44. 7:48

    If it's looking at some code and that code has, let's say, some documentation, it-- the documentation needs to live in the comments so if it ever grabbed into that code, it reads the comment, goes to that file, finds all the information about it.

45. 8:01

    That's just one example. The second is close the loop, right? You've gotta make a self-healing system 'cause slop is inevitable. There is going to be some slop that's going to seep in, but you need to have a pipeline and a way to close the loop to remove the slop, to detect it, and to be able to, like,

46. 8:17

    self-heal the system. And then you need to iterate continuously, and I can't emphasize this enough. Like, you can't assume that you do this for a month and you're done.

47. 8:26

    Like, things are gonna change constantly underneath, so you need to keep this as one of the things that you have to do as an organization. And the third most important thing is treat it like a human problem, guys.

48. 8:36

    Like, this isn't-- it's not, you know, oh, it's this tool. Like, people will figure it out. Let's just mandate our way through life. Like, that's just not gonna work.

49. 8:44

    So treat it like a human problem. Fear is real. Human emotions are real. We should recognize it.

50. 8:50

    So, you know, enough gyaan or as, like-- It's more like the Hindi way to say, like, you know, " Enough prof--" Like, I'm kinda giving you sermons, but, like, how do you really do this, right?

51. 9:00

    Like, these are our principles. What's the real playbook? So here's what we did, and here's, like-- I'm not gonna over, like, emphasize on this is the exact way to do it, but this is, like, roughly how we did it, and you can take from it what you choose.

52. 9:14

    The first thing is do the basics, right? You gotta do them right. Progressive disclosure, I can't emphasize this enough, is really, really powerful, right? Find your best ICs and find what-- how they're making the code base work for them.

53. 9:27

    Take those practices and pass them org-wide. People can't live in their own practices, and this is really hard for engineers to do. It's basically accepting that my setup is imperfect and, like, engineers don't like to hear that.

54. 9:39

    But you gotta figure out a way to find those best practices and ship them across, make sure that that's kind of a shared setup.

55. 9:47

    The second thing we did was there's one high-value skill that we invested in. In our case, it was this thing called ShipIt. What it did was the moment you're done with your code, it takes care of everything from code done to PR ready for review, which means you gotta open a PR, figure out your opinions, handle the--

56. 10:05

    handle all the comments, you know, handle all the PR descriptions, the merge comments, everything, right? It handles CI failures. It basically runs through these loops. And what this meant was often the skill was running for over an hour, and that scared people.

57. 10:19

    But once you actually figure out, once they saw the value, they get invested, right? Because it's one skill which tells them, "Okay, this AI thing can actually work for me.

58. 10:28

    I don't need to constantly babysit it. I can trust it."

59. 10:32

    The third thing, and really important, is to close the loop, right? So we wired issues and boards into the repo. Like, we added CI/CD. We added agentic reviews. We have, like, a code gardener that actually goes back and looks through a whole bunch of, like-- Every night it'll run and look at the code and check if something,

60. 10:46

    like, not organized correctly. What does correct organization mean will depend on your code base.

61. 10:53

    Get people invested, and I can't emphasize this enough. You have to win over the skeptics. It's really easy to say, like, the skeptic is just someone who's scared. It's really hard to get them to buy in, but if you can get them to buy in, you know you're doing something right.

62. 11:07

    You have to get them to be able to edit and play with the shared setup 'cause that's the true way you know that they're actually invested, right? And this is where you gotta it- ensure you're iterating constantly.

63. 11:19

    If people are-- Like, and this is the hardest thing for engineers, again, 'cause you're basically saying, "I'm never gonna, like, get to perfection in my setup." But you've gotta be okay with that.

64. 11:29

    You have to do it, and you have to treat it like, you know, X percent of your IC time is probably gonna be spent on iterating on this thing, which is not gonna lead to, like, meaningful PRs, like, upfront, but it's useful and it's worth it.

65. 11:42

    And, you know, I don't wanna say this is perfect, right? Like, we faced a ton of issues while doing this, and I'm just gonna walk you through some of them.

66. 11:49

    But it's an iteration loop, so you've gotta treat it like a piece of feedback. So what are the problems we hit, right? There are too many issues. Like, when we started, like, we blew up to, like, four or five hundred issues, uh, without, like, you know, just I think within, like, a couple of weeks, which is a

67. 12:03

    crazy number for, like, a repo. And then, you know, there's so many, like, different agents all trying to create issues 'cause they've not been wired correctly. There's a lack of agreement.

68. 12:12

    People-- You know, as soon as people saw like, "Oh, this isn't working perfectly or the way I expected it," they're-- it's super easy for them to say, "You know what?

69. 12:19

    I'm just gonna go back to babysitting my agent." You don't want that. You wanna actually take their feedback and put it back into the skill and improve the skill.

70. 12:28

    Agents are taking too long. This is actually one of those expectation-setting things. It's good if agents take too long. That means you can actually go off and do other things, and you have confidence that they're doing the right thing.

71. 12:38

    At the end of the day, like, the moment we hit this reasoning paradigm, the longer the agent, like, thought, the better its output. You can create like a s-- It's almost like a similar mindset for your entire code base and for your skills.

72. 12:51

    There's going to be merge hell, and, like, we just have to deal with it. We have to figure out a way to deal with this.

73. 12:57

    There is going to be slop when you're gonna write experiments. Treat it like its own thing, right? What we said was, "Okay, people are generating this code, but it's not relevant.

74. 13:06

    It's not gonna be shipped." It's just an-- It's a prototype. Treat it like one. Get it to opt out of all the rigorous other standards you've got across your code base.

75. 13:15

    And realize people vary on the spectrum, right? And depending on the day, depending on, like, what they're going through, they're gonna vary on the spectrum. You have to be able to talk to them and figure out, "Hey, okay, why, why are you facing this?"

76. 13:27

    Like, is it if the model changed, the harness changed? Again, you need to go revisit something. Figure that out.

77. 13:34

    And I think the biggest, like, the easiest way to say this is instead of saying like, "The model is so dumb," like, we have to ask, "How can I make it smarter?"

78. 13:42

    Or, "How can I edit?" Like, and not... This is why I've, like, crossed out the my. It's not a personal setup. It's the shared setup that you have to invest in.

79. 13:51

    And it's a mindset, right? You have to go full send.

80. 13:55

    And, like, I wanna end you-- I wanna end with this. Like, I learned skiing, like, a couple years back, and the fir-- like, the hardest thing for me was you actually have to commit to it.

81. 14:05

    If you're pizza breaking, you're gonna crash, like, no matter what. You have to commit to the speed in order to actually get and feel like, "Okay, that's how I can turn, and that's how I can truly ski."

82. 14:15

    And so I'm gonna leave you with this. Like, just be okay with fa-- be okay with failing. You have to, you know, go full send and be okay with falling.

83. 14:24

    It's fine. The point is to be able to recover from that, and that'll allow you to truly feel the AGI.

84. 14:32

    Yeah. Well, that's, that's me, and I'm happy to take any questions. [audience applauding]

85. 14:41

    Yep. Strategies that you found best for progressive disclosure. Uh, so the-- I'm gonna repeat the question for the recording. Uh, strategies that you found best for progressive disclosure. So I think a couple things, right?

86. 14:53

    Uh, the first thing is, even in your skill.md files, don't overload it. Like, we've kinda set a hard limit for, like, a hundred lines in your skill.md 'cause your skill is really a folder.

87. 15:04

    So that's step one. Make sure, and I think I spoke about this during the talk, but

88. 15:09

    when you have, like, some code that requires, like, a runbook, make sure the runbook is reflected in the comments so that if a code-- if somehow the code, like, the agent figures its way into, like, grepping into the code base and find that file, it knows I need to go look at this for, like, all the description

89. 15:26

    of how this is relevant, right? You have to organize. And I think this is why I talk about harness engineering because your entire code base can be set up to encourage progressive disclosure.

90. 15:36

    Don't overload your Claude.md or your agent's.md file into, like, one big thing. You wanna make sure that, like, it's a thin index that can point through the right files, and that's what the agent gets in its, like, first prompt 'cause that's what gets loaded when it starts to work.

91. 15:50

    So these are, like, some really powerful strategies that... And the way you know this is working is when you give it a prompt. When you give it the first prompt, see what it's doing.

92. 15:59

    Is it grepping, or does it know where to go? Or how much context is it burning when it, like, immediately? So is it like... I think, like, twenty, twenty-five K tokens get taken anyway.

93. 16:08

    But, like, how much more is getting added? If you're coming to, like, forty K, fifty K, like, something's wrong. That's not really progressive disclosure. So you have to figure out these boundaries, and then, like, based on this, it's an iteration cycle.

94. 16:31

    All right. Well, if there, there aren't any other questions, feel free to find me. Happy to talk about harness engineering in general or, like, anything else. But yeah, thank you for listening. [outro music]