Software Development Agents: What Works and What Doesn't

Talk transcript

1. 0:00

   [on hold music] Today, I'm gonna talk a little bit about, uh, coding agents and, uh, how to use them effectively, really.

2. 0:20

   Um, if you're anything like me, you found that, uh, you found a lot of things that work really well and a lot of things that, uh, don't work very well.

3. 0:27

   Um, so a little bit about me. Uh, my name is Robert Brennan. I've been building, uh, open source development tools for, for over a decade now. Uh, and my team and I, uh, have created, uh, an open source, uh, software development agent called OpenHands, formerly known as OpenDevin.

4. 0:45

   So to, to state the obvious, in twenty twenty-five, software development is changing. Uh, our jobs are, are very different now than they were two years ago, uh, and they're gonna be very different two years from now.

5. 0:56

   Uh, and the thing I wanna convince you of is that coding is going away. Uh, we're gonna be spending a lot less time actually writing code. Uh, but that doesn't mean that software engineering is going away.

6. 1:05

   Uh, we're paid not to, to type on our keyboard, but to actually think critically about the problems that are in front of us. Uh, and so if we do AI-driven development correctly, um, it'll mean we spend less time actually, like, leaning forward and squinting into our IDE and more time kinda sitting back in our chair and thinking,

7. 1:22

   you know, what does the user actually want here? Uh, what are we actually trying to build? What, what problems are we trying to solve as an organization? Uh, how can we architect this in a way that sets us up for the future?

8. 1:32

   Uh, the AI is very good at that, at that inner loop of development, the write code, run the code, write code, run the code. It's not very good at those kind of big picture tasks that have to take into account, um, that have to, like, empathize with the end user, uh, take into account business level objectives, uh,

9. 1:46

   and that's where we come in as, as software engineers.

10. 1:52

    Uh, so let's talk a little about, uh, what actually a, a coding agent is. Uh, I think this word agent gets thrown around a lot these days. Uh, the meaning has started to, to drift over time, uh, but at the core of it is this, this concept of agency.

11. 2:04

    Um, it's this idea of, of taking action out in the real world. Um, and these are, these are the main tools of a software engineer's job, right? We have a, a code editor to actually modify our code base, navigate our code base.

12. 2:17

    Uh, you have a terminal, uh, to help you actually run the code that you're, that you're writing. Uh, and you need a web browser in order to look up documentation and maybe copy and paste some code from Stack Overflow.

13. 2:27

    So these are kind of the core tools of the job, and these are the tools that we give to our agents to let them do their whole, uh, development loop.

14. 2:35

    I also wanna contrast, uh, you know, coding agents from some more tactical codegen tools that are out there. Um, you know, we kinda started a couple years ago with things like, uh, GitHub Copilot's auto-complete feature where, you know, it's literally wherever your cursor is pointed in the code base right now, it's just filling out two or three

15. 2:51

    more lines of code. Um, and then over time, things have gotten more and more agentic, more and more asynchronous, right? Uh, so we got, like, AI-powered IDEs that can maybe take a few steps at a time without, uh, a developer interfering.

16. 3:04

    And then, uh, now you've got these tools like Devin and OpenHands where you're really giving an agent, you know, one or two sentences describing what you want it to do.

17. 3:12

    It goes off and works for five, ten, fifteen minutes on its own and then comes back to you with a solution. This is a much more powerful way of working.

18. 3:19

    You can get a lot done. Uh, you can send off multiple agents at once. Um, you know, you can focus on communicating with your coworkers or goofing off on Reddit while these agents are, are working for you.

19. 3:31

    Um, and it's, uh, it's just-- it's a, it's a very different way of working, but it's a much more powerful way of working.

20. 3:38

    Uh, so I wanna talk a little bit about how these agents work under the hood. I feel like, uh, once you understand what's happening under the surface, uh, it really helps you build an intuition for how to use agents effectively.

21. 3:50

    Uh, and at its core, um, an agent is this loop between a large language model and the, and the external world. So, uh, the large language model kinda serves as the brain, uh, and then we have to repeatedly take actions in the external world, get some kind of feedback from the world, and pass that back into the

22. 4:06

    LLM. Um, uh, so basically at every, every step of this loop, we're asking the LLM, "What's the next thing you wanna do in order to get one step closer to your goal?"

23. 4:15

    Uh, it might say, "Okay, I wanna read this file. I wanna make this edit. I wanna run this command. I wanna look at this webpage." Uh, we go out and take that action in the real world, get some kind of output, whether it's the contents of a webpage, uh, or the output of a command, and then stick

24. 4:28

    that back into the LLM for the next turn of the loop.

25. 4:33

    Uh, just to talk a little bit about kind of the core tools that are at the agent's disposal. Uh, the first one, again, is a, is a code editor.

26. 4:40

    Um, you might think this is, this is really simple. It actually turns out to be a fairly, uh, interesting problem. Uh, the naive solution would be to just, like, give the old file to the LLM, uh, and then have it output the entire new file.

27. 4:51

    That's not a very efficient way to work though. If you've got a thousand line, uh, thousand line of-- thousands of lines of code and you wanna just change one line, uh, you're gonna waste a lot of tokens printing out all the lines that are staying the same.

28. 5:03

    So most, uh, contemporary, um, agents use, uh, like a, a find and replace type editor or a diff-based editor to allow the LLM to just make tactical edits inside the file.

29. 5:15

    Uh, a lot of times they'll also provide, uh, like an abs- abstract syntax tree or some kind of way to allow the agent to navigate the code base more effectively.

30. 5:25

    Uh, next up is the terminal, and again, you would think text in, text out should be pretty simple, but there are a lot of questions that pop up here.

31. 5:31

    You know, what do you do when there's a long-running command that has no standard out for a long time? Do you kill it? Do you let the LLM wait?

32. 5:37

    Uh, what happens if you wanna run multiple commands in parallel, run commands in the background? Maybe you wanna start a server and then run curl against that server. Uh, lots of really interesting, uh, problems that crop up, uh, when you have an agent interacting with the terminal.

33. 5:51

    Uh, and then probably the most complicated tool is the web browser. Again, there's a naive solution here where you just-- uh, the agent just gives you a URL, and you give it a bunch of HTML.

34. 6:00

    Um-

35. 6:00

    That's, uh, very expensive because there's a bunch of cruft inside that HTML that the, the LLM doesn't really need to see. Uh, we've had a lot of luck passing it, uh, accessibility trees or converting to Markdown and passing that to the LLM, um, or allowing the LLM to maybe scroll through the webpage if there's a ton of

36. 6:15

    content there. Um, and then also if you start to add interaction, things get even more complicated. Uh, you can let the LLM, uh, write JavaScript against the page, uh, or we've actually had a lot of luck basically giving it a screenshot of the page with labeled nodes, and it can say what it wants to click on.

37. 6:31

    Uh, this is an area of active research. Uh, we just had a contribution about a month ago that doubled our accuracy on web browsing. Uh, I would say this is, uh, this is definitely a space to watch.

38. 6:43

    Uh, and then I also wanna talk about, about sandboxing. Uh, this is a really important thing for agents because if they're gonna run autonomously for several minutes on their own without you watching everything they're doing, you wanna make sure that they're not doing anything dangerous.

39. 6:56

    Uh, and so all of our agents run inside of a Docker container by default. Um, they're, they're totally separated out from your workstation, so there's no chance of it running rm -rf on your home directory.

40. 7:07

    Um, increasingly though, we're giving agents access to third-party APIs, right? So you might give it, uh, access to a GitHub token or access to your AWS account. Super, super important to make sure that those credentials are tightly scoped and that you're following, uh, the principle of least privilege as you're granting agents access to do these things.

41. 7:27

    All right, I wanna move into some best practices.

42. 7:31

    Uh, my, my biggest advice for folks who are just getting started is to start small. Um, the best tasks are things that can be completed pretty quickly, you know, a single commit, uh, where there's a clear definition of done.

43. 7:42

    You know, you want the agent to be able to verify, okay, the tests are passing. I must have done it correctly. Um, or, you know, the merge conflicts have been solved, et cetera.

44. 7:51

    Um, and tasks that are easy for you as an engineer to verify, uh, were done completely and, and correctly. Um, I like to tell people to start with small chores.

45. 7:59

    Uh, very frequently you might have a pull request where there's, you know, one test that's failing or there's some lint errors or there's merge conflicts, uh, bits of toil that you don't really like doing as a developer.

46. 8:08

    Those are great tasks to just shove off to the AI. They tend to be, tend to be very rote. Uh, the AI does, does them very well. Um, but as your intuition grows here, as you get used to working with an agent, you'll find that you can give it bigger and bigger tasks.

47. 8:20

    Uh, you'll, you'll understand how to communicate with the agent effectively. Um, and I would say for, for me, for my co-founders, and for our, for our biggest power users, uh, for me, like 90% of my code now goes through the agent, and it's only maybe 10% of the time that I have to drop back into my IDE

48. 8:35

    and kinda get my hands dirty in the code base again.

49. 8:40

    Uh, being very clear with the agent about what you want is super important. Uh, I specifically like to say, you know, you need to tell it not just what you want, but you t- need to tell it how you want it to do it.

50. 8:48

    You know, mention specific frameworks that you want it to use. Uh, if you want it to do like a test-driven development strategy, tell it that. Uh, mention any specific files or function names that it can, that it can go for.

51. 8:59

    Um, this not only, uh, helps it be more accurate and, uh, you know, more clear as to what exactly you want the output to be, um, it also makes it go faster, right?

52. 9:08

    It doesn't have to spend as long exploring the code base if you tell it, "I want you to edit this exact file."

53. 9:14

    Um, this can save you, uh, a bunch of time and energy, and it can save, uh, a lot of, a lot of tokens, a lot of actual like inference costs.

54. 9:22

    Uh, I also like to remind folks that in an AI-driven development world, code is cheap. Um, you can throw code away. You can, you can experiment and prototype. Uh, I love if I, if I have an idea like on my walk to work, I'll just like, uh, you know, tell OpenHands with my voice like, "Do X, Y,

55. 9:38

    and Z," and then when I get to work I'll, I'll have a PR waiting for me. 50% of the time I'll just throw it away. It didn't really work.

56. 9:44

    50% of the time it looks great and I just merge it and it's, and it's awesome. Um, it's, uh, it's really fun to be able to just rapidly prototype using AI-driven development.

57. 9:53

    Um, and I would also say, you know, if you, if you try to, try to work with the agent on a particular task and it gets it wrong, maybe it's close and you can just keep iterating within the same conversation and it's already built up some context.

58. 10:05

    If it's way off though, just throw away that work. Start fresh with a new prompt based on, uh, what you learned from the last one. Um, it's really, really, uh, I think, uh, it's a new, new sort of muscle memory you have to develop to just throw things away.

59. 10:18

    Sometimes it's, uh, hard to throw away tens of lines, tens of thousands of lines of code that, uh, have been generated 'cause you're used to that being a very expensive, uh, bunch of code.

60. 10:28

    Uh, these days it's, it's very easy to kind of just start from scratch again.

61. 10:34

    This is probably the most important bit of advice, uh, I can give folks. Uh, you need to review the code that the AI writes. Uh, I've seen more than one organization run into trouble, uh, thinking that they could just vibe code their way to a production application, uh, and just, you know, automatically merging everything that came out

62. 10:49

    of the AI. Um, but, uh, if you just, you know, don't review anything, you'll find that your code base just grows and grows with this tech debt. You'll find duplicate code everywhere.

63. 11:00

    Uh, things get out of hand very quickly. Uh, so make sure you're reviewing the code that it outputs and make sure you're pulling the code and running it on your workstation or running it inside of an ephemeral environment, uh, just to make sure that, you know, the agent has actually solved the problem that you asked it to

64. 11:12

    solve. Uh, and I like to say, you know, trust but verify. You know, as you work with agents over time, you'll build an intuition for, for what they do well and what they don't do well, and you can generally trust them to, to, um, you know, operate the same way today that they did yesterday.

65. 11:28

    Um, but you really, you really do need a human in the loop. Um, you know, one of our big learnings, uh, with OpenHands, in the early days if you opened up a pull, pull request with OpenHands, uh, the, that pull request would show up as owned by OpenHands.

66. 11:42

    It would be the little hands logo, uh, next to the pull request, uh, and that caused two problems. One, it meant that the human who had triggered that pull request could then approve it and basically bypass our whole code review system.

67. 11:53

    You didn't need a second human in the loop to, uh, before merging. Uh, and two, oftentimes those pull requests would just languish. Uh, nobody would really take ownership for them.

68. 12:02

    Uh, if there was, like, a failing unit test, nobody was, like, jumping in to make sure the test passed. Um, and those, they would just kind of, like, sit there and not get merged.

69. 12:10

    Or if they did get merged and something went wrong, the code didn't actually work, we didn't really know who to go to and be like, you know, "Who caused this?"

70. 12:16

    There was nobody we could hold accountable for that breakage. Um, and so now if you open up a pull request with OpenHands, your face is on that pull request, you're responsible for getting it merged, you're responsible for any breakage it might cause down the line.

71. 12:30

    Well, and then, uh, I do wanna just close just by going through a handful of use cases. Uh, this is always kind of a tricky topic because agents are great generalists.

72. 12:37

    They can, they can hypothetically do anything as l- as long as you kind of, like, break things down into bite-sized steps that they can take on. Um, but in that, in that, um, in the spirit of starting small, I think there are a bunch of use cases that are, like, really great day one use cases for agents.

73. 12:53

    My favorite is resolving merge conflicts. This is, like, the biggest chore as a part of my job. Uh, OpenHands itself is a very fast-moving code base. Uh, I can say there's probably no PR that I make that, uh, I get away with zero merge conflicts.

74. 13:06

    Um, and I love just being able to jump in and say, "Add OpenHands, fix the merge conflicts on this PR." Uh, it comes in and, you know, it's such a rote task, it's usually very obvious, you know, what changed before, what changed in this PR, what's the intention behind those changes, and OpenHands knocks this out, you know,

75. 13:20

    99% of the time. Uh, addressing PR feedback is also a favorite. Uh, this one's great because somebody else has already taken the time to clearly articulate what they want changed, and all you have to do is say, "Add OpenHands, do what that guy said."

76. 13:34

    Uh, and again, uh, like you can see in this example, uh, OpenHands did exactly what this person wanted. I don't know React super well, and, uh, our front-end engineer was like, "Do X, Y, and Z," and he mentioned a whole bunch of buzzwords that I don't, I don't know.

77. 13:47

    OpenHands knew all of it, and, uh, was able to address his feedback exactly how he wanted.

78. 13:53

    Uh, fixing quick little bugs. Um, you know, you can see in this example we had, uh, an input, uh, that, you know, was a text input, should've been a number input.

79. 14:00

    Uh, if I wasn't lazy, I could've, like, dug through my code base, found the right file. Um, but it was really easy for me to just, like, quickly-- I think I did this one from directly inside of Slack.

80. 14:09

    Uh, just add OpenHands, fix this thing we were just talking about. Uh, and, uh, it's just, you know, really, I don't even have to, like, fire up my IDE.

81. 14:18

    Um, it's just, it's a really, really fun way to work.

82. 14:22

    Uh, infrastructure changes I really like. Uh, usually these involve looking up some, like, really esoteric syntax inside of, like, the Terraform docs or something like that. Um, OpenHands and, you know, the underlying LLMs tend to just, like, know, uh, the right Terraform syntax, and if not, they can, they can look up the documentation using the browser.

83. 14:39

    Um, so this stuff is, uh, is really great. Sometimes we'll just get, like, an out of memory exception in Slack and immediately say, "Okay, OpenHands, increase the memory."

84. 14:48

    Uh, database migrations are another great one. Uh, this is one where I find, uh, I often leave best practices behind. I won't put indexes on the right things. I won't set up foreign keys the right way.

85. 14:58

    Uh, the LLM tends to be really great about following all best practices around database migrations. So again, it's kind of like a rote task for developers. It's not very fun.

86. 15:07

    Um, uh, the LLM's great at it. Uh, fixing failing tests, uh, like, on a PR. Uh, if you've already got the code 90% of the way there, there's just a unit test failing because there was a breaking API change, very easy to call in an agent to just clean up the, the failing tests.

87. 15:23

    Uh, expanding test coverage is another one I love because, uh, it's a very, um, safe task, right? As long as the tests are passing, it's, uh, generally safe to just merge that.

88. 15:33

    So if you notice a spot in your code base where you're like, "Hey, we have really low coverage here," just ask, uh, ask your agent to, uh, expand your test coverage in that area of the code base.

89. 15:42

    Uh, it's a great quick win, uh, to make your code base a little bit safer.

90. 15:47

    Then everybody's favorite, building apps from scratch. Um, you know, I would say if you're shipping production code, again, don't just, like, vibe code your way to a production application.

91. 15:56

    Uh, but we're finding increasingly internally at our company, a lot of times there's, like, in a little internal app we wanna build. Uh, like, for instance, we built a way to, uh, debug OpenHands trajectories, debug OpenHands sessions.

92. 16:08

    Um, uh, we built, like, a whole web application that since it's just an internal application, we can vibe code it a little bit. We don't really need to review every line of code.

93. 16:15

    It's not really facing end users. Uh, this has been a really, really fun thing for our business to just be able to churn out these really quick applications, uh, just to serve our own internal needs.

94. 16:25

    Um, so yeah. Uh, greenfield is a great, great use case for agents. Um, that's all I've got. Uh, would love to have you all join the, the OpenHands community.

95. 16:33

    You can find us on GitHub, allhandsai/openhands. Um, join us on Slack, Discord. Uh, we'd love to build with you. [upbeat music]