Devin 2.0 and the Future of SWE

Talk transcript

1. 0:00

   [upbeat music] Yeah. Well, thank you guys so much for having me.

2. 0:16

   It's exciting to be back. It's, uh, I, I was last here at AI Engineer one year ago. Um, and it's kinda crazy. I, I've always been-- I, I've been telling Swix that we need to have these conferences way more often if it's gonna be about AI software engineering.

3. 0:28

   Probably should be, like, every two months or something like that [chuckles] with the pace that everything's done. But, but, but gonna be fun to, to talk a little bit about, um, you know, what we've seen in the space and, and what we've learned over the last twelve or eighteen months, uh, building Devin over this time.

4. 0:43

   And I wanna start this off with, um, Moore's law for AI agents. And so you can kinda think of the, the, the capability or the capacity of an AI by how much work it can do uninter-uninterrupted until you have to come in and step in and intervene or steer it or whatever it is, right?

5. 1:02

   And, um, you know, in GPT-3, for example, it's if you were to go and ask GPT-3 to do something, you know, it could probably get through a few words or so, and then it'll say something where it's like, "Okay, you know, this is probably not the right thing to say." [chuckles]

6. 1:14

   Um, and GPT-3.5 was better, and GPT-4 was better, right? Um, and, and so people talk about these lengths of tasks, and what you see in general is that that doubling time is about every seven months, which already is pretty crazy, actually.

7. 1:27

   But in code, it's actually even faster. It's every seventy days, which is two or three months. And so, you know, if you look at various software engineering tasks that start from the simplest single functions or single lines, and you go all the way to, you know, we're doing tasks now that take hours of humans' time, and, and

8. 1:46

   an AI agent is able to just do all of that, right? Um, and if you think about doubling every seventy days, I mean, basically, you know, every two to three months means you get four to six doublings every year.

9. 1:56

   Um, which means [chuckles] that the amount of work that an AI agent can do in code goes something between sixteen and sixty-four x in a year, every year, at least for the last couple years that we've seen.

10. 2:08

    Um, and it's kinda crazy to think about, but, but it-- that sounds about right, actually, for, for what we've seen. You know, eighteen months ago, I would say the only really-- the only product experience that had PMF in code was just tab completion, right?

11. 2:21

    It was just, like, here's what I have so far. Predict the next line for me. That was kind of all you really could do, um, in, in a way that really worked.

12. 2:29

    And we've gone from that, obviously, to full AI engineer that goes and just do-does, does all these tasks for you, right, and implements a ton of these things. And people ask all the time, what is the, um, you know, what, what, what is the, the future interface or what is the right way to do this or what

13. 2:46

    are the most important capabilities to solve for? And I think funnily enough, the answer to all these questions actually is it changes every two or three months. Like, every time you get to the next tier, the, the, the bottleneck that you're running into or the most important capability or the right way you should be interfacing with it,

14. 3:02

    like, all these actually change at, at each point. And so I wanted to talk a bit about some of the, the, those tiers for us over the last year or so.

15. 3:13

    Um, and, you know, over the course of that time, obviously, you know, when, when we got started, um, in the end of twenty twenty-three, obviously, agents were not even a concept.

16. 3:20

    Um, and now everyone has, you know, everyone's talking about coding agents. People are doing more and more and more. Uh, and, and it's very cool to see. Um, and, and each of these has kind of been almost a, a discrete tier for us.

17. 3:31

    Um, and so right, right around a year ago when we were doing the last AI Engineer talk actually, um, the, the biggest use case that we really saw that, that was getting broad adoption was what I'll kind of call these repetitive migrations.

18. 3:43

    And so I'm talking, like, JavaScript to TypeScript or, like, upgrading your Angular version from this one to that one or going from this Java version to that Java version or something like that.

19. 3:54

    Um, and those, those kinds of tasks in particular, what you typically see is you are--

20. 4:01

    you, you have some massive code base that you wanna apply this whole migration for. You have to go file by file and do every single one, and usually, the set of steps is pretty clear, right?

21. 4:10

    If you go to the Angular we-website or something like that, it'll tell you, "All right. Here's what you have to do, this, this, this, this, this." And, um, you wanna go and execute each of these steps.

22. 4:19

    It's not so routine that there, you know, there's no classical deterministic program that solves that. But there's kind of a clear set of steps, and if you can follow those steps very well, then you can do the task.

23. 4:29

    And, you know, this was the thing for us because that was all you could really trust agents to do at the time. You know? You could do harder things once in a while, and you could do some really cool stuff occasionally.

24. 4:40

    But as far as something that was consistent enough that you could do it over and over and over, um, these kinds of, like, repetitive migrations that you would be doing for, you know, ten thousand files were, you know, in, in, in many ways, the, the, the easiest thing.

25. 4:52

    Which was cool actually because it, it, it was also kind of the, the most annoying thing for humans to do. [chuckles] And I think that's generally been the trend where, um, AI has always done these more boilerplate tasks and the more tedious stuff, the more repetitive stuff, and we get to do the, the, the more fun, creative stuff.

26. 5:08

    Um, and obviously, as time has gone on, it's, it's taken on more and more of that boilerplate. But for a problem like this one, a lot of what you need to do is you need Devin to be able to go and execute a set of steps super reliably.

27. 5:22

    And so a lot of this was, you know, I would say the big capabilities problems to solve was mostly instruction following. And so we built this system called Playbooks where basically you could just outline a very clear set of steps, have it follow each of those step-by-step, and then do exactly what's said.

28. 5:37

    Now, if you think about it, obviously, a lot of software engineering does not fall under the category of literally just follow ten steps step-by-step and do exactly what it said.

29. 5:44

    But migration does. And it allowed us to go and actually do these, and, and this was kind of, I would say, the first big use case of Devin that really, um, that really came up.

30. 5:53

    I think one of the other big systems that got built around that time, which we've since rebuilt many times, is knowledge or memory, right? Which is, you know, if you're doing the same task over and over and over again, then often the human will have feedback on, "Hey, by the way, you have to remember to do X

31. 6:08

    thing," or, "You have to, you know-- you, you need to do Y thing every time."

32. 6:13

    When you see this, right? Um, and so basically an ability to, to just maintain and understand the learnings from that and use that to improve the agent in every future one.

33. 6:22

    And those were kind of the, the big problems of the time. You know, and that was summer of last year. And around end of summer or fall or so, you know, I think the, the, the kind of big thing that started coming up was as these systems got more and more capable, instead of just doing the most

34. 6:38

    routine migrations, you could do, you know, these more s- still pretty isolated, but, but, but, but a bit broader of these general kind of bugs or features where you can actually just tell it what you want to do and have you- have it do it, right?

35. 6:50

    And so for example, "Hey Devin, in this, uh, repo select dropdown, can you please just list the currently selected ones at the top? Like, having the checkboxes throughout is, it's just doesn't really..."

36. 7:00

    And, and Devin will just go and do that, right? And so if you think about it, it's,

37. 7:04

    y- you know, it's, it's, it's something like the kind of a level of task that you would give an intern.

38. 7:09

    And there are a few particular things that you have to solve for, um, with this. First of all, usually these, these, these changes are pretty isolated and pretty contained, and so it's one, maybe two files that you really have to look at and change to do a task like this.

39. 7:23

    But at least you do still need to be able to set up the repo and work with the repo, right? And so you wanna be able to run lint, you wanna be able to run CI, all of these other things, so you know, at, to at least have the basic checks of whether things work.

40. 7:36

    One of the big things that we built around then was the ability to really set up your repository, uh, ahead of time and build a snapshot, um, that, that you could start off, that you could reload, that you could roll back and all of these kinds of primitives as well, right?

41. 7:48

    So having a, this clean remote VM that could run all these things, it could run your CI, it could run your linter, uh, and, and so on. Um, but that's when we started to really see, I would say, a bit more broader value, right?

42. 8:00

    I mean, migrations is one particular thing, and for that particular thing, we were showing a ton of value, and then we started to see where, you know, with these bug fixes or things like that, you would be able to just generally get value from Devin as, as almost like a junior buddy of yours.

43. 8:14

    And then in fall, things really moved towards just much broader bugs and requests. And here it's, you know,

44. 8:23

    most, most changes, again, you know, where you, you jumping another order of magnitude. Most changes don't just contain themselves to one file, right? Often you have to go and look, see what's going on.

45. 8:31

    You have to diagnose things. You have to figure out what's happening. You have to work across files and make the right changes. Often these changes are, you know, hundreds of lines.

46. 8:39

    If it's like, "Hey, I've got this bug, let's figure out what's going on, let's solve it," right?

47. 8:44

    And you know, there, there are a lot of things here that, that really started to make sense and really started to be important, but, but one in particular I'll just point out was there's a lot of stuff that you can do with not just looking at the code as text, but thinking of it as this whole hierarchy,

48. 8:58

    right? So, so understanding call hierarchies, running a language server, uh, is a big deal. You have Git commit history, which you can look at, which informs how, how these different files relate to one another.

49. 9:09

    You have, um, um, obviously you have, like, your linter and things like that, but, but you're able to kind of reference things across files. And so, like, one of the big problems here, I think, was, uh, kind of working with the context of it and getting to the point where it could make changes across several files.

50. 9:25

    It could be consistent across those changes. It would be able to understand across the codebase. And here was really the point, I would say, where you started to be able to just tag it and have it do an issue and just have it build it for you.

51. 9:36

    Um, and so Slack was a, it was, you know, a, a huge part of the workflow then. Um, and, and, and it was just,

52. 9:42

    it, it, it, it made sense because it's where you discuss your issues and it's where you set these things up, right? So you would tag Devin in Slack and say, "Hey, by the way, we've got this bug.

53. 9:49

    Please take a look." Or, you know, "Could you please go build this thing?" Uh, this is especially fun part for us because this is right around when we went GA, uh, and a lot of that was because it was, it got to the point where you truly could just get set up with Devin and ask it a

54. 10:03

    lot of these broad tasks and, and just have it do it. Um, but, but a lot of these, you know, a, a lot of the work that we did was around having Devin have better and better understanding of the codebase, right?

55. 10:13

    And if you think about it, you know, from the human lens, it's the same way, where on your first day on the job, for example, being super fresh on the codebase, it's kinda tough to know exactly what you're supposed to do.

56. 10:22

    Like, a lot of these details are things that you understand over time or that a, a representation of the codebase that you build over time, right? Um, and Devin had to do the same thing and had to understand, "How do I plan this task out before I solve it?

57. 10:34

    How do I understand all the files that need to be changed? How do I go from there and make that diff?"

58. 10:41

    And around spring of this year, um, again, every, every gap is, like, two or three months. You know, the, we, we got to an interesting point, which is once you start to get to harder and harder tasks, you as the human don't necessarily know everything that you want done at the time that you're giving the task, right?

59. 10:59

    If you're saying, "Hey, you know, I, I'd like to go and, um, improve the architecture of this," or, you know, "This, this function is slow. Like, let's, let's profile it and look into it and see what needs to be done."

60. 11:09

    Or, "Hey, like, you know, we really should, should handle this, this error case better, but, like, let's look at all the possibilities and see what we should, you know, what the right logic should be in each of these," right?

61. 11:19

    And basically what it meant is that this whole idea of taking a two-line prompt or a three-line prompt or something and then just having that result in a, a Devin task was, was not sufficient, and you wanted to really be able to work with Devin and specify a lot more.

62. 11:34

    And around this time, along with this kind of, like, better codebase intelligence, um, we had a few different things that, that, that, that came up, and so we released DeepWiki, for example.

63. 11:42

    Um, and the whole idea of DeepWiki was, you know, funnily enough, is Devin had its own internal representation of the codebase, but it turns out that for humans it, it was great to look at that too, to be able to understand what was going on or to be able to ask questions quickly about the codebase.

64. 11:56

    Um, closely related to that was with Search, which was the ability to really just ask questions about a codebase and understand, um, some, some piece of this. A- and a lot of the workflow that really started to come up was actually basically this, this more iterative workflow where the first thing that you would do is you would

65. 12:12

    ask a few questions. You would understand. You would basically have a more L2 experience where you can go and explore the codebase with your agent.

66. 12:20

    Figure out what has to be done in the task, and then set your agent off to go do that. Because for these more complex tasks, you kind of needed that, right?

67. 12:29

    Um, and so, so, you know, that was a, I, I would say, kind of like a big paradigm shift for us then is, is understanding, you know, this is what also came along with Devin 2.0, for example, and the in IDE experience where often, yeah, you wanna be able to have points where you closely monitor Devin for

68. 12:43

    10% of the task, 20% of the task, and then have it do, uh, work on its own for the other 80, 90%.

69. 12:51

    Um, and then lastly, most recently in June [laughs] which is now, uh, it was kind of, yeah, the really the ability to just truly just kill your backlog and hand it a ton of tasks and have it do all these at once.

70. 13:02

    And, you know, if you think about this task, in, in many ways, I would say it's, it's almost like a culmination of, of many of these different things that, that had to be done in the past.

71. 13:09

    You have to work with all of these systems, obviously. You have to integrate into all these. Certainly, you wanna be able to, to work with Linear or with Jira or systems like that.

72. 13:15

    But- Right ... you have to be able to scope out a task to understand what's meant by what's going on. You have to decide when to go to the human for more approval or for questions or things like that.

73. 13:25

    You have to work across several different files. Often you have to understand even what repo is the right repo to make the change in if, if your, if your org has multiple repos, or what part of the code base is the right part of the code base that needs to change.

74. 13:37

    Um, and to really get to the point where you can go and do this more autonomously,

75. 13:42

    first of all, um, you, you have to have, like, a, a really great sense of confidence, right? And so, um, you know, rather than just going off and doing things immediately, you have to be able to say, "Okay, I am quite sure that this is the task, and I'm gonna go execute it now," versus, "I don't understand

76. 13:57

    what's going on. Human, please give me help," [laughs] basically, right? But, but the other piece of it is

77. 14:04

    this is, I think, the era where testing and this asynchronous testing gets really, really important, right? Which is if you want something to just deliver entire PRs for you for tasks that you do, especially for these larger tasks, you wanna know that it is ha- can, can test it itself, and often the, the agent actually needs this

78. 14:20

    iterative loop to be able to go and do that, right? So it needs to be able to run all the code locally. It needs to know what to test.

79. 14:26

    It needs to know what to look for. Um, and in many ways, it's just a, a much higher context problem to solve for, right, is this testing itself.

80. 14:34

    And that brings us to now. And obviously it's a, it's a pretty fun time to see because now what we're thinking about is, hey, maybe if instead of it doing it just one task, it's, you know, how, how do we think about tackling an entire project, right?

81. 14:46

    And after we do a project, you know, what, what goes after that? A- A- A- And maybe one point that I would just p- make here is

82. 14:53

    we talk about all these 2Xs, you know, that happen every couple months. And I think from a kind of cosmic perspective, all the 2Xs look the same, right? But in practice, every 2X actually is a different one, right?

83. 15:04

    And so when we were just doing, you know, tab completion, line, single line completion, it really was just a text problem. It is just, like, taking the single file so far and just predict what the line is next, right?

84. 15:15

    Over the last year or year and a half, we've had to think about so much more. How do you, how do you work with the human in Linear or Slack or Jira?

85. 15:22

    How do you take in feedback or steering? Um, how, how do you help the human plan out and do all these things, right? And moreover, obviously, there's a ton of this, the tooling and the capabilities work that have to be done of how does, how does Devin test on its own?

86. 15:35

    How does Devin, um, uh, you know, make a lot of these longer term decisions on its own? How does it debug its own outputs or, or run the right shell commands to figure out what the feedback is, uh, and go from there?

87. 15:47

    And so it's super exciting now that there's a lot more, uh, there's a lot more coding agents in the space. It's, uh, it's, it's very fun to see, and I think that, you know, we- we're [laughs] gonna see another 16 to 64 X over the next twelve months as well.

88. 15:59

    And, uh, and, and so, yeah, sup- super, super excited.

89. 16:03

    Awesome. Well, that's all. Thank you guys so much for having me. [audience applauding] [upbeat music]