The emerging skillset of wielding coding agents

Talk transcript

1. 0:00

   [upbeat electronic music] My name is, uh, Beyang.

2. 0:16

   I'm the CTO and co-founder of a company called Sourcegraph. Uh, we build developer tools. And today, I wanna share with you some of the observations and insights that we've had on the sort of like emerging skill set of how to wield coding agents.

3. 0:30

   That sound good to everyone? Yeah. All right. Cool. Um, okay, so let's check in on the, uh, the, the agent discourse. Uh, I don't know if you all saw this, but a couple of days ago, uh, there were some spicy tweets, uh, about the, the efficacy of AI coding agents or, you know, inefficacy, depending on your perspective.

4. 0:49

   So, um, Jonathan Blow, who's a really talented developer, he, uh, basically single-handedly coded up the indie game Braid, if you're familiar with that. So like he's, he's kinda like god tier status in terms of coding ability, um, retweeted Alex Albert, who is also someone I respect and admire a lot, uh, works at Anthropic, basically claiming that, you

5. 1:09

   know, all the hype around coding agents and code AI in general, uh, was just-- it, it's just hype, right? There's no substance there. Um, and then there were some responses, and there was kind of a spectrum of responses too.

6. 1:20

   You know, we had some other, uh, big names in the developer world like Jesse Frizelle. She was one of the early, uh, contributors, maintainers of Docker. She's also really legit.

7. 1:30

   Uh, she said basically s-something to the effect of like, um, "I think you're right, uh, but you're in like the top .01% of programmers, Jonathan. For the rest of us, you know, down here in this room that aren't on Mount Olympus, uh, it actually helps a lot."

8. 1:44

   Um, but not super helpful if you're, if you're really, really good. Um, but then we also had folks like Eric, uh, S. Raymond, who is like the one of the fa- fathers of open source, uh, who had a very spicy reply.

9. 1:56

   He's basically like, "Look, I, you know, consider myself to be pretty decent at programming, and, uh, these things help a lot." Uh, and then, uh, the, the kinda-- my favorite one of this was actually the, the, the f- top of Hacker News post, uh, that was written by, um, uh, Thomas, uh, uh, Tachek, uh, who is a,

10. 2:15

    a really legit security engineer. Um, some of you may have seen this, uh, trending. It was-- He's basically taking the opposite view of like, you know, there's some really smart people there who are very AI skeptical, but they're nuts.

11. 2:25

    Like, uh, th-these things are really useful. Um, so I, I'm guessing if you're at this conference, you probably lean toward coding agents, uh, are substantively, uh, useful and there's something there.

12. 2:36

    I don't know. Uh, just a, just a guess. But I think even within this room, there's probably a spectrum of, uh, best practices and opinions about like where agents are good, uh, you know, w-whether they're restricted to like small edits or like front-end applications or weekend vibe coding, whether they actually work on your production code base.

13. 3:00

    And, um, I think this is just, uh, indicative of, of the dynamic technical landscape that, that we're in right now. And a couple of months back, I, I read this blog post from this [REDACTED:gender], Jeff Huntley.

14. 3:10

    So Jeff was a, a senior engineer at Canva at the time, and, uh, his role at Canva is really interesting. He basically went around interviewing all the, uh, developers inside of Canva using AI tools like Cursor and, and other things and seeing how they're using it.

15. 3:26

    And he basically came to the conclusion that like most people were holding it wrong, uh, which is, which is really interesting. And he ca- he came up with a blog post about like all the different anti-patterns that he was seeing.

16. 3:36

    Um, but my summation of, of, of that blog post is like the, the number one mistake that people are using with coding agents right now is they're trying to use coding agents the same way they were using AI coding tools, uh, six months ago.

17. 3:51

    Um, and, and therefore they're wrong, which is kinda crazy because normally if you're, you know, using a tool, uh, it-- the, the best practices don't change in, in six months.

18. 4:02

    Typically, the things that you learn that are good, uh, will still be like present, uh, and, and, uh, you know, topical and relevant six months, uh, down the line.

19. 4:11

    But I think we're in a really interesting moment in time right now. And you know, why the sudden change? I think it's because, uh, of this step function transition, uh, that we've experienced in model capabilities in the past six months.

20. 4:25

    So, you know, we've, we've all been around since the dawn of generative AI, the ancient year of, uh, 2022, right? November 2022 was when, uh, ChatGPT launched, right? And every year, uh, now, you know, this is now the year three, th- you know, three after ChatGPT, right?

21. 4:44

    We're now living in the AI future. Um, but I think there's already been kinda like three distinct waves or, or eras largely driven by the evolution of frontier model capabilities.

22. 4:55

    Um, and the model capabilities really dictate the, uh, ideal architecture, uh, that, that becomes dominant at the application layer. So in the GPT-3 era, all the models were text completion models, uh, which meant all the applications that people were building, uh, were these like copilots or autocomplete tools.

23. 5:13

    So the dominant u- uh, UX paradigm was like you type some stuff, it types some stuff, you type some more, and that's how you would interact. Uh, and then ChatGPT came along, uh, with GPT-3.5 w- which was instruct tuned to interact like a, uh, a chatbot.

24. 5:28

    Uh, and suddenly people realized like, oh, it's not just completing the next thing I'm talking about. I can actually ask it questions like I can a human now. Uh, and then some other people came along, uh, we were part of this crowd, w-we realized like, hey, um, you know what's even better than just like asking it questions?

25. 5:43

    You can actually copy-paste stuff into the chat and say like, "Here is some code from my code base. Use that as an example and pattern match against that." And you-- That helps it generate, you know, a little bit better code or less, uh, less fake code or less hallucinated code than, uh, it did before.

26. 5:59

    And, uh, that basically meant that everyone at the application layer was building a RAG bot, uh, in, in 2023. So like a chatbot plus a, a RAG retrieval engine.

27. 6:08

    But now, uh, I think we've entered a, a new era, and I don't- I'm not sure if everyone realizes it or maybe this is kind-- I, I don't know.

28. 6:16

    Like, who agrees with this statement? Like, who thinks it's a real paradigm shift? Yeah. Okay. And then who's, who here is like, "Ah, that's a bunch of bullshit"? Anyone? [laughs]

29. 6:24

    Feel free to... I like... Okay, okay. So maybe I'm s- maybe I can just skip this slide. Um,

30. 6:31

    so we're, we're now living in the era of agents, and the new model capabilities, uh, really dictate a, a new application architecture. And so one of the things that we asked ourselves at Sourcegraph is, you know, a lot of the existing tools in the market that were designed for the era of GPT-4 and, uh, Claude 3.

31. 6:47

    So they... A lot of the application stuff, uh, features and UX and UI was really built around the capabilities or in some cases the limitations of the chat-based LLMs.

32. 6:57

    Um, and so if we're going to design a coding agent from the ground up to unleash the capabilities of tool using LLMs, agentic LLMs, uh, what would that look like?

33. 7:09

    Okay, so here are my spicy takes. Uh, these are controversial design decisions that I think are, are better to make in the age of agents. Uh, and many of these go against the best practices that kind of emerged in the chatbot era.

34. 7:23

    Okay, so number one is, uh, the agent should just make edits to your files. It shouldn't ask you at every turn like, "Hey, you know, I wanna make this change.

35. 7:32

    Should I apply it?" Uh, if it's asking you and it's wrong, uh, it's already done the wrong thing and it's wasted your time. Uh, humans need to get, uh, more out of the inner loop and more kinda like on top of the loop.

36. 7:43

    Like still steering it and guiding it, but less, you know, micromanaging and, and managing every change. Second thing is, do we still need a, a thick client, uh, to, uh, to manipulate the LLMs?

37. 7:56

    Like, do we still need a fork VS Code? That's like the salty way of saying this, right? Um, the VS Code fork became the, the culmination of the AI coding application I think for, for the, the chatbot era.

38. 8:07

    But there's this question of like, you know, if the contract of an agent is you ask it to do stuff and then it does stuff, do you really still need all that UI built around like context management and applying the proposed change in the code base, or can you just ask it to do stuff and expect it

39. 8:21

    to, to do the right thing? Third, I think we're gonna move beyond the choose your own model, uh, phase. So I think in the chatbot era it was very easy to swap models in and out and you're like, "Oh, you know, a new model came along.

40. 8:34

    Let me swap it out and see how well it attends to the context that my retrieval engine fetches." Um, in the agentic world, there's a much deeper coupling, uh, because the LLM that you're using essentially becomes the, the brains of these agentic chains, and so it's much harder to rip and replace.

41. 8:50

    And I think a lot of people in this room who have tried mixing and matching, uh, you know, different models, uh, in the context of agents have found that it, you know, swapping out a different model and expecting similar results is, is very different.

42. 9:01

    A lot of the like, a lot of the LLMs out there aren't even good at the basics of tool use yet, so it's, it's very difficult to just replace the brains.

43. 9:10

    Um, four is I think we're gonna move past the era of fixed pricing. Uh, agents eat up a lot of tokens and so they look expensive relative to chatbots.

44. 9:20

    Uh, but the comparison that more and more people are making is how much human time is it saving? So they're still cheap relative to, to human time saved. And the fixed pricing model actually introduces a perverse incentive now where, uh, it's like selling gym memberships, right?

45. 9:34

    Like if, if I sold you a membership to my chatbot and you're now paying me, you know, 20 bucks a month, uh, my incentive now is to push the inference cost as low as possible, and the easiest way to do that is to use dumber models.

46. 9:45

    Um, but dumber models just waste more of your, more of your time. Um, sorry, this is a long list. Um, uh, hopefully it's not too tedious but, um, I, I think these are important points.

47. 9:57

    Uh, the, the second to last point I'll make is I think the Unix philosophy is gonna be more powerful here than vertical integration. So in developer tools, the ability, uh, to use simple tools in ways that compose well with other interesting tools is really powerful.

48. 10:11

    And so I think especially with agents where there's less of a need to create like a lot of UI around it, you're gonna start to see more command-driven tools, command line tools and, and things like that.

49. 10:21

    Um, and then last but not least is, uh, you know, we had an existing, uh, RAG chat coding assistant. Maybe some of you have used it. It was called Cody.

50. 10:30

    Um, it still exists. We're still supporting it. Uh, it's still in heavy use across, you know, many Fortune 500 companies. Uh, but we decided to build a, a new application from, from the ground up for the agentic world because we didn't wanna be constrained by all the assumptions and constraints that, uh, we, we built into the application

51. 10:49

    layer, uh, for the previous generation of LLMs.

52. 10:53

    And one analogy I like to draw here is, you know, what, uh, the, the, the early days of the internet, right? Like in the early days of the internet the, the way people, you know, jumped into the, the web was using an interface on the left.

53. 11:08

    This was before like most people knew what the internet was about, what it was capable of, and that was the right interface for the first generation of the internet because like what can you do with the internet?

54. 11:15

    Well, like there's a bunch of different things. You can look at like trending celebrities. You can, you know, buy automobiles. You can look at movie reviews. All these things you might not have thought of, and so it's, it's useful to have in front of you.

55. 11:26

    But at some point it gets a little tedious like clicking through all the different hyperlinks and navigating your way through, and then the, the real power of the web was sort of unleashed by just like the one simple text box where you just like type what you're looking for and, and you get to it.

56. 11:41

    And I think, you know, with, with agentic UIs, that's what, uh, we should be striving for, both in developer tools and in a lot of different application paradigms. Okay, so what does that look like in practice?

57. 11:53

    So w- when we went to design this thing, um, our coding agent is called Amp, uh, and Amp has two clients and this is what they look like. So both are like very, very bare bones.

58. 12:03

    A lot of people, you know, look at this and like, "What is this? It's just a text box. What, what can I do with it?" Um, and, and that was by design, you know, that, the, be- for all the reasons I just mentioned.

59. 12:13

    Um- One client is just a simple VS Code extension, um, that allows us to take advantage of some nice things that you get in, in VS Code, like being able to view diffs.

60. 12:22

    That's really important in the agentic coding world. I, I often joke that, like, that's now... I use that view more than the editor view now. Um, and, and the second was a CLI.

61. 12:32

    So, uh, just stripping things down to bare bones. It has access to all the same tools as the, the VS Code extension does, but it's just something that you can invoke in your command line.

62. 12:40

    You can also script it, compose it with other tools.

63. 12:45

    Okay. So, uh, what, what, what does this actually look like in practice? Um, I, I wanna do something a little bit risky here, which is, um, in the past I've done a lot of like, you know, "Hey, here's me building a simple app," like those sorts of demos.

64. 12:59

    But I actually wanna show off, like, where we think this is most useful which is like, "Hey, I'm working on an application that has real users. Let me actually make a contribution to that code base given all-- w- with all the, like, existing constraints."

65. 13:12

    And so I actually wanna... Uh, I'm just gonna code a little bit. Well, I don't even know how far we're gonna get. Um, but this is, this is Amp.

66. 13:19

    Uh, th- this is VS Code running Amp in the sidebar, and it's open to the Amp code base. Um, and what I wanna do is implement, like, a simple, uh, change, uh, to this application.

67. 13:31

    So the change that I'm gonna make is Amp has a server component, and the server exists, uh, as a way to provide the LLM inference point. It also provides, like, team functionality.

68. 13:43

    We have a way to share, like, what different teams are doing and what different users are doing with AI, so you can kinda learn from other users. There's a leaderboard.

69. 13:51

    It's fun. Um, but there's also these things called connectors which allow Amp to talk to external services. So our issue tracker is Linear. Um, and so I've integrated Linear, uh, into Amp here, but I'm kind of annoyed because it's using this generic, like, network icon, and I would really like to customize this icon such that when you

70. 14:07

    plug in the Linear MCP endpoint, it, it uses a more appropriate icon, like a, a checkbox or something issuey.

71. 14:14

    Um, so I've already filed this as a Linear issue, and I'm just gonna ask, uh, "Can you find the Linear issue about customizing the Linear connector icon? Uh, then implement it."

72. 14:30

    So what this will do is, um, it has access to a set of tools. Um, I can go over here to the tool panel and see what tools it has access to.

73. 14:39

    Some are local, some are built in. Um, it's got the standard tools like read and edit file, run Bash command. Uh, you can also plug in things like Playwright and Postgres via MCP.

74. 14:49

    Uh, and then Linear is also plugged in, uh, through this, so i- we're basically talking to the Linear API, uh, through the MCP server.

75. 14:57

    And, uh, what this will do is it will use the Linear, uh, issues API, um, and it will search issues. It found tw- 50 issues, and the one that I was referring to is at the top here.

76. 15:09

    So add a special icon for the Linear connector. Uh, and now it's gonna go and implement, uh, the thing for me. Um, and one thing to note here is it's just making these tool calls on its own.

77. 15:22

    I'm not asking it, uh, to use specific tools. Um, we've also tried to make the, uh, information that you see, uh, minimal. So, like, you don't need to see all the API tool calls that it's making underneath the hood or, like, crowd out the transcript with a bunch of things.

78. 15:38

    Most of the time, uh, w- we just wanna keep it simple because the contract we wanna provide to users is like the, uh, the, the feedback loops here are more robust, and you don't have to, um, micromanage this as much.

79. 15:50

    Another thing I wanna point out here is the search tool that this is using is, is actually a sub-agent. So it's actually spinning off a sub-agent loop that uses a form of agentic search that has access to a bunch of different search tools, uh, keyword search, uh, uh, just regular graph, uh, looking up file names.

80. 16:09

    Uh, if you wanna inspect what it's doing, it-- you can click the expand thing and see, like, what different path it's taking, what files it's reading, what things it uncovered.

81. 16:17

    Uh, but again, by default, we think this is, like, an implementation detail, and hopefully it should just surface the, the right thing. Um, so it's, it's working. It's gathering context.

82. 16:26

    Um, another thing I wanna call out in this interface is, um, as we've gotten more feedback, we've k- we've kind of designed this thing to be more multi-threaded. So there's a quick keyboard shortcut that allows you to, like, quickly tab through the different threads that you're running, and it's a common paradigm in, in our user community to

83. 16:41

    be running more than one of these things at a time. Um, and it takes a little bit used to get, get used to the, uh, the context switching. Like, developers hate context switching, right?

84. 16:50

    Like, we like to be, uh, i- in, in flow, in, in focus. Um, w- typically what we see here is, um, the, the secondary thread will either be something that's, like, a lot shallower so that you can quickly page back to the main thread.

85. 17:03

    Or what I like to do is, while the agent is working, I actually like to understand the code, uh, at a deeper level myself so I can better understand what, what it's gonna do.

86. 17:11

    So, uh, I could ask something like, "Can you show me how connectors and connections work in Amp?" You can ask it to draw a picture, uh, of that.

87. 17:25

    So we'll kick that, kick that thread off in parallel. We'll check back in on what this [REDACTED:gender] is doing. So it's found, uh, it's read a bunch of files.

88. 17:32

    It's read f- some front-end files. Our front end is written in Svelte. Um, and as you can see, it's, it's being fairly thoughtful about reading the appropriate files before it actually goes and, and does the work, and we find that this is really important, uh, to make the feedback cycles, uh, more robust.

89. 17:48

    Um, otherwise, the, the anti-pattern is you just, like, get into the weeds of, like, steering it manually.

90. 17:54

    Um, it's also got this to-do list thing at the bottom, uh, that helps it structure and plan out the, the longer term tasks so that it doesn't go, like, immediately dive into the code.

91. 18:04

    This is a classic mistake that, like, human developers make too, where you, like, dive into the code too early, and then you get lost in the weeds, and then it takes a while to dig yourself out.

92. 18:12

    Um, okay, so it's making some changes. Um-

93. 18:16

    One other thing that I like to point out here is, you know, I mentioned that I use the diff view in VS Code now probably more than the editor view.

94. 18:24

    Uh, VS Code actually has a really nice diff view. I have it hotkeyed, um, so I can open it up quickly. And most of the, my time in VS Code now is spent just, like reviewing the changes that it, uh, it makes.

95. 18:35

    And I actually like this a lot better than, uh, like GitHub PRs or, uh, Git diff on the command line, just 'cause it's in the editor, you can see the whole file and, uh, jump to definition, uh, even works.

96. 18:46

    Um, so yeah. We'll, we'll just wait a little bit for it to, to do its thing. I actually think it's, it's probably made...

97. 18:55

    Looks like it's getting, it's getting there. Um, let's... It's probably just running, like tests. Let's see if we go back here, if it's updated the icon at all.

98. 19:10

    Okay. So hasn't gotten there yet, but I think it's on the right track.

99. 19:17

    Does it write its own tests? Uh, the question was, does it write its own tests? Yes, it typically writes its own tests, and if, if it doesn't, you can prompt it to, to do so.

100. 19:26

     So, uh, it's doing a lot of things. It's reading a lot of files. It's making these edits incrementally and then checking the diagnostics. Um, and then now let's see if it works.

101. 19:35

     Okay, cool. So you see here the icon has been updated, and this is without me really steering it in, in any fashion. Um,

102. 19:44

     notice here on this page that this icon didn't update though. Um, and so this is actually not surprising to me because this change, as many changes in production code bases are, often more nuanced than it seems at the surface.

103. 19:56

     So in this case, the reason it's not getting it here is because, uh, this is the admin page, and the piece of data we need to know, uh, we need to read in order to tell that this is a linear MCP, uh, rather than a generic MCP, is actually part of the config.

104. 20:13

     We have to look at the endpoint of the MCP URL. And in order to do that, you have to read the config, but the config might also contain secrets.

105. 20:20

     Doesn't con-contain secrets in this case, but might contain secrets in other cases. So we actually prohibit those secrets from being sent to non-admin pages. Um, so it's not surprising to me that, like the first pass, it didn't get that right, but let's see if it can get...

106. 20:31

     Like I'll just nudge it a little bit. So like, uh, I noticed that the icon changed on admin connections, but not on settings. Um,

107. 20:45

     can you investigate why? And, uh, in the interest of time, we'll check back on this later. How about that? Uh, we'll, we'll let it run and we'll, we'll see if, if it, it can find its way to the right solution there.

108. 21:00

     Um, is it okay if I go a little bit over since, uh, we started a little bit... Okay, cool. Is it okay with you all if I go a little bit over?

109. 21:07

     Okay. Are you still having fun? Yeah. Okay, cool. So that was like a brief demo of just, like the interaction patterns and, and the UI. We try to keep it really minimal.

110. 21:17

     Um, we've released this to like a, a small group so far. Uh, the, the sign-up is now publicly open. It's been open for about two weeks, but we haven't done a lot of, like marketing around it.

111. 21:26

     And, and that's kind of been intentional because we're really trying to design this for where we think the, the puck is going. And so we've, we've done a lot to curate this community of people who are trying to experiment with LLMs and figure out, like how the interaction paradigms are gonna change over the next six to 12

112. 21:43

     months. And so our user community is really people who are like spending nights and weekends, uh, a lot of time with this thing to see what they can get it to do.

113. 21:52

     And so actually, one of the, the most insightful things, and actually the main topic o- of this talk is lessons that we've learned from just like looking at what our power users are doing and seeing what interesting behavior patterns, uh, th- that they're kinda like implementing.

114. 22:07

     Um, and so like the, the average spend, uh, for agents is, is growing. It's a lot more than the average spend was for chatbots or, or autocomplete. But one oth-other interesting thing that we've noticed among the user base is that, uh, there's a huge variance in terms of how much people use this thing.

115. 22:25

     Um, uh, to the point where like there, there's like an upper echelon of users that are spending like thousands of dollars per month, uh, uh, just in inference costs.

116. 22:35

     And at first we're like, "This has gotta be abuse," right? Like someone out there is, you know, poked, uh, you know, found some way to exploit the inference endpoint, is, is using it to power some like [REDACTED:origin], uh, you know, AI girlfriend or whatever.

117. 22:48

     But actually, no. When we, when we spoke to, uh, the people using it, we actually found that they were doing real things, and we're like, "Hmm, that's interesting. What the hell are you doing?"

118. 22:56

     Um, and from those insights and the conversations, we basically, uh, have encapsulated a series of like best practices or emergent, um, like power user patterns, uh, for how the, the very, you know, uh, dominant users, the, the most active users are, are using this thing.

119. 23:14

     And this has informed our, our product design process as well. So one of the, the first changes that we made, um, was we noticed that a lot of the power users were ver- writing very long prompts.

120. 23:25

     It was not like the simple, kinda like Google style, like three keywords and just like, uh, read my mind and expect something good to happen. Uh, they actually wanted to write a lot of detail because they realized that LLMs are actually quite programmable.

121. 23:37

     If you give them a lot of context, they will follow th- those instructions and get further than if you just give them like a one-line sentence. And so we made the default behavior of the Enter key in the AMP input just new line.

122. 23:49

     So you have to hit Command Enter s- to submit. And this throws a lot of the new users off because they're like, "Wait a minute. Why doesn't it just enter?"

123. 23:54

     Like, you know, i- if I'm in like Cursor or whatever, it's just Enter. That's easy. That's intuitive. But actually, what we want to push users to do is to write those longer prompts because that actually yields better results, and I think that's one of the things that prevents people, uh, who are still in the kinda like chat

124. 24:09

     LLM, uh, mode from, from unlocking some of the, you know, cool stuff that, that agents can do.

125. 24:17

     Um, another thing that people do very intentionally is direct the agent to look at a relevant context and feedback mechanisms. So, you know, context was very important in the chatbot, uh, era.

126. 24:29

     It's still important in the agentic era. Now, agents do have a good, good amount of, like, built-in knowledge for how to use tools to acquire context. Like, you saw that before when it was using the search tool to find different things.

127. 24:40

     Um, and, and it was executing the test to, and, and, uh, linters to see if, uh, the code was valid. Um, but there's still some cases, especially in production code bases, where it's like, "Oh, we do things in a very specific way that are kinda, like, out of distribution."

128. 24:55

     And, and so, like, some, like, less, uh, less agentically inclined users at that point will just give up. They're like, "Ah, you know, agents aren't capable of working with, like, back-end code yet."

129. 25:05

     But what we've noticed is the power user are like, "Actually, let me try to just tell it how to run, you know, the build in this particular subdirectory, run the tests," and that helps it complete the feedback loop so that it can get the validation to get further.

130. 25:19

     Um, feedback loops are gonna be a big theme, uh, of, of this talk. So ano- another, like, uh, dominant, uh, paradigm here is constructing these, like, front-end feedback loops.

131. 25:28

     So, like, a, a really common formula is you have the Playwright MCP server, and then there's, uh, a thing called Storybook, which is basically a way to encapsulate or componentize, uh, a lot of your front-end components.

132. 25:39

     It makes it very easy to test individual components without loading your entire app. And, you know, you probably should've been doing this anyways as a human developer because you get a fast feedback loop.

133. 25:48

     You make a change, see it reflected instantly, you get the auto reload, and then go back to your editor. But with agents, y- you, you kinda notice it more because you're no longer, like, in the weeds doing the thing.

134. 25:57

     You're like, "Oh," you're a-- You're almost like the developer experience engineer for your agent. It's like, "How can I make it loop back faster?" And so what the agent will do is, like, you know, make the code change, use Playwright to open up the page in the browser, snapshot it, uh, uh, and then loop back on itself.

135. 26:12

     And it does that via Storybook because it's much faster than reloading the, the entire app.

136. 26:17

     So you, you put Playwright as a tool for you? Yes. So it's, um, one of the default recommended tools. Okay. So it's right here. Um, and actually it looks like, looks like that run completed.

137. 26:30

     I wonder if, uh... It looks like it did approximately the right thing. Um, sorry, just to jump out of the slides here a little bit. So now you can see, like, the icon is, is customized on the settings page, not just the admin page.

138. 26:44

     And if you look at how it did that, I think it did the right thing. So if you look at the diff, um, it actually looked at the surrounding code and was like, "Oh, there is an existing mechanism for plumbing non-secret parts of the config through to the UI.

139. 26:57

     Let me kinda, like, use that as a reference point," and it actually plumbed exactly that, like, uh, field through to the front end. So now if I add, like, additional fields to the MCP config that do contain secrets, uh, it'll-- This is, like, whitelisted, so it'll still only send the endpoint URL over to the client.

140. 27:15

     You know, what, basically what it needs to make that icon customization. Um, so yeah, I know, like, you know, it's not a super impressively visual change, but, like, a lot of such changes in, in messy production code bases are like that, and it's cool to see the agent, uh, be able to tease, tease out that nuance.

141. 27:34

     Um, okay. I know we're a little bit over time. Can I-- People mind if I keep going or... Uh, okay, cool. I can leave. [laughs]

142. 27:44

     Um, there's, uh, some additional, uh, tips and tricks. Most of this talk is just, like, sharing what we've learned from our power users. So another thing that we've noticed is, like, there's this kinda, like, this prevailing narrative that, like, you know, agents are gonna make programmers lazy.

143. 27:56

     It's gonna make it so we don't really understand what, uh, what, what's going on in the code, so we're gonna ship more slop. But we've actually found the inverse happen, uh, with, with the power users.

144. 28:05

     They're actually using agents to better understand, uh, the code. And so it is a really good onboarding tool. Like, we just hired this [REDACTED:gender], Tyler Bruno. He's a very precocious young developer.

145. 28:15

     He's actually still in college, but he's working full time in addition to taking classes. Uh, so really bright but also, you know, a bit green. Um, he's been using Amp to just, like, quickly ramp up on how the different pieces connect together.

146. 28:28

     Uh, and it can draw diagrams and, and point you to specific pieces of the code and, uh, it's really good at accelerating that. And then a corollary to this is, like, you know, we all do a form of onboarding to new code whenever we, we do a code review.

147. 28:39

     Like, by definition, code review is, is new code, and oftentimes it's new code that contains blogs or is hard to understand or is a bit of a slog. Um, and so rather than just, you know, ignore, uh, the code that the AI generates and just commit it blindly, uh, we find that our user base is actually using

148. 28:57

     this tool to do more thorough code reviews. So, like, I've adopted this practice myself where if I have to review a very large diff, the first thing I do is ask the agent to consume the diff and generate a high-level summary so I can have, like, a high-level awareness.

149. 29:10

     And then I ask it, like, "Hey, if you were a smart senior dev, wha- what's the entry point into this PR?" 'Cause, like, often half the time, uh, half the battle is just, like, finding the right entry point.

150. 29:20

     And, uh, psychologically, I, I often put off code reviews because I'm like, "Oh, it's gonna be a pain and it's gonna take forever just to, like, figure out where I should start reviewing it, so I'll just do it tomorrow."

151. 29:30

     But this thing just, like, it helps lower that activation energy and, and make code reviews more thorough and, and actually, uh, dare I say, like, a little bit fun and enjoyable now.

152. 29:40

     Um, subagents are also a thing. So, uh, we implement s- uh, the search tool as a subagent, uh, in the very beginning, but we're seeing more and more, uh, use cases emerge for subagents.

153. 29:50

     And the, the general best practice with subagents, uh, is that they often are useful for longer, uh, more complex tasks because the subagent is-- allows you to essentially preserve the context window.

154. 30:00

     So, like, the, the, the, the quality of the LLM will degrade, uh, over time. You know, Sonnet 4 has a context window of 200K, but we see degradation typically around, like, 120 or 130K, and by the time you get, hit 170 tokens, uh, it, you see, start to see more kinda, like, off the rails and crazy behavior.

155. 30:19

     Uh, but subagents allow you to encapsulate, uh, the context used up by a specific subtask, like implementing a small feature, uh, such that it doesn't pollute the, the main agent.

156. 30:30

     Okay, so th- that was a quick tour of, of, uh, a lot of best practices. Just to recap, like, the anti-practices. Uh, the common anti-patterns are just, like, micromanaging the agent.

157. 30:39

     Like, using it like you would a chatbot, where you have to kinda, like, steer it at every interaction or rev- review every edit it's making. Um, another common, uh, anti-pattern is just, like, under-prompting, so not giving it enough detail.

158. 30:51

     Like, LLMs, their knowledge comes from two places. It either comes from their training data or from the context that you give it. And so, uh, you know, it's fine if you do a five-word prompt if you're coding up, like, a 3D Flappy Bird game from scratch because that's well represented in the training set.

159. 31:08

     They're really good at that. They're trained to do that. Um, but if you're trying to make a subtle nuanced change to your large existing code base, you should be giving it all the details that you would give a colleague on the team, uh, uh, to point them in the right direction.

160. 31:23

     And then last but not least, like, agents are, are not a, a vehicle to, like, TLDR the code. If anything, they're the opposite. You should be using them to do much more thorough code reviews more quickly.

161. 31:33

     Uh, the human is still... You're, you're ultimately responsible for the code that you ship, and you shouldn't view this as a human replacement. It's really a tool that you can wield to make yourself, uh, 10, 100x more effective.

162. 31:45

     Uh, last tidbit. So one of the things that we've noticed among the very, very, very top 1% of the 1% is, uh, this, this, uh, inclination to run multiple of these things in parallel.

163. 31:58

     So the, uh... Jeff Huntley, who wrote that blog post, uh, that I, I showed earlier, um, he started putting out these, uh, Twitter, uh, streams. They're, they're about, like, four hours long each, and it's basically just, uh, what he's...

164. 32:12

     He's working on, like, a compiler on the side, and what he does is he, he, uh, constructs prompts for, like, three or four different agents to, to work on different parts of, of the compiler.

165. 32:23

     Uh, and he's gotten to the point where he's prompting it such that he feels confident enough in the feedback loops where he just, like, hits Enter, lets them run, and then goes to sleep.

166. 32:31

     And then, like, this thing just runs on Twitter for a while, and I think he's doing this to kinda, like, spread the word. It's like, "Hey, you can use this for serious engineer- uh, engineering."

167. 32:39

     Like, compilers are not some, like, vibe coding, uh, vibe coded weekend project. They're, they're, they're real tech. They're, they're difficult to build. Um, and it is possible to use agents for, for code like this, but it has to be a very intentional skill that, that you, you practice.

168. 32:56

     And so I think it's cool. I think, like, there's a lot of people thinking in terms of, like, agent fleets and where the, the world is going, but I, I do think that the way that we'll actually get there is by, by building these, like, composable building blocks that allow people like Jeff to go and, like, combine

169. 33:09

     them and, and, uh, come up with interesting UIs. I think this is just running in, like, Tmux or some window manager.

170. 33:16

     Okay, so, like, the, the takeaways I just wanna leave you with is, one, you know, contrary to what some might say, and, you know, look, there's a lot of smart senior developers out there who think AI's over-hyped, and maybe parts of it are.

171. 33:27

     But, like, I think coding agents are very real, and it is, uh, I think a high ceiling skill. It's like I think we will probably invest in learning how to use these things in the same way that we invest in learning how best to use our editor or our programming language of choice.

172. 33:44

     And I think the only way you can learn this stuff is, is by doing it and then sharing it out with others. Uh, and one of the reasons we built the kinda, like, thread sharing mechanism into Amp is to help encourage knowledge dissemination so that, like, if you discover an interesting way of, of using it, you can

173. 33:58

     share that out with your team. Um, but yeah, that's it. If you wanna kinda, like, see a, a recap of the best practices in this talk, we've actually put out, like, a, an Amp owner's manual that guides new users how to, to best use it.

174. 34:11

     Um, I'll also be around afterwards. We have a booth in the main expo hall. Uh, I'm supposed to say, too, if you stop by the booth we'll give you, like, $10 in, in free credits.

175. 34:20

     So if anything you saw here was of interest to you and you wanna try this out, um, stop by and, and say hi. [audience applauds]

176. 34:32

     I notice you still, uh, type, uh, can you, and then you correct your- [laughs] ... uh, typos, which I guess you said you shouldn't do. Yeah, I- Is that habit?

177. 34:40

     I... It's, it's part habit and it's part paranoia that in, like, a live demo setting there will be some, uh, typo token that will trigger off the rails behavior.

178. 34:49

     But it... Like, I think that was more of a concern that I learned in, like, 2023 when it actually mattered. 'Cause, like, these days LLMs are, are more and more, like, typo robust I would say. [outro music]