The Making of Devin

Talk transcript

1. 0:00

   [upbeat music] I'm Scott from Cognition AI, and I'm gonna tell you guys a little bit about, um, you know, the, the, the early makings of Devin.

2. 0:20

   We're still super, super early on, and, and also a little bit about kind of the space as a whole and, and what's coming next. Um, and I thought it'd be nice to, to start with a demo first.

3. 0:29

   Um, uh, it sounds like some of you guys have, have already seen some of the videos, but I brought a nice custom one here for the World's Fair today.

4. 0:35

   Um, so I'll just show that quickly. And here, I basically said, "Hey, Devin..." This was this morning, by the way. This was a huge scramble. I, I said, "Hey, Devin, uh, I want you to build a mobile-friendly website to play the name game."

5. 0:47

   So I have a lot of trouble memorizing names and faces. I don't know about you guys. Um, but I basically just said, you know, "Here's a, here's a TSV file of a bunch of names and faces."

6. 0:57

   These are all the speakers, uh, here at the World's Fair this week. And I said, "Can you set up the game so that you show two different random faces and then show the names of one of them and have me guess which one is which," right?

7. 1:08

   And I gave kind of a few instructions on how the game should work.

8. 1:12

   And so Devin is a fully autonomous software engineer, and what that means is Devin has access to all the same tools that a human software engineer would have when it was building-- w-when, when they were building something like this.

9. 1:23

   And so the first thing that Devin's gonna do is Devin's gonna make a plan. Um, and you can see here, you know, kind of a basic plan coming out.

10. 1:30

    Um, one of the interesting things about this is the plan changes a lot over time, and so, you know, as you get new information or new feedback, you update your plan accordingly with that too.

11. 1:40

    After that, Devin's basically just running this the same way that a human would. And so if you can take a look, you know, Devin makes a new directory for the name game website, starts a new React app, you know, all, all the same primitives.

12. 1:50

    Um, works on building it out and building the code. You know, reads the TSV file to take a look, uh, at what's going on here. Um, and is just kinda generally working through it and jumping through.

13. 2:00

    Um, it comes out and deploys this first version after, after some minutes, and I'll just pull this up quickly. So that's what this looks like. Um, it's, it's close, but not quite there, right?

14. 2:09

    I mean, it shows the-- It's still showing the names, and I think maybe I didn't quite specify that exactly. Um, but, you know, you can click, uh, the name and it got that correct.

15. 2:17

    Um, and so I just went ahead and just gave it some more feen- feedback in plain English. And so I said, "Hey, you know, can you hide the two names until I click on the answer?

16. 2:24

    And also, can you probably restyle the Play Again button?" It's like, you know, somehow it's a, it's a little off on this page. Um, and I kept going, uh, and just, just kinda gave it more and more feedback over time.

17. 2:34

    And I also asked it, "Hey, can you add a streak counter as well? You know, can you, can you keep track of how many I got correct and, you know, reset to zero?"

18. 2:41

    Um, you know, a few of these other things. Um, and the website it ultimately deployed, uh, was this one right here. And so this is Justine, for example. Keeps track of my streak, and you can see it's kind of, uh, ramping it up.

19. 2:53

    And so, you know, if I were to, for example, um,

20. 2:57

    if I got this one wrong on purpose, then you would see the streak would reset to zero, you know, and it would go on. And so I actually played this game and, and learned the names of everyone, which was, which was super helpful, by the way. [laughs]

21. 3:07

    And, uh, you know, you guys can play it too. It's, uh, it's right here if you wanna try it out. This has all the speakers. I think it was something like a hundred and seventy speakers here at the, at the World's Fair, um, this week.

22. 3:16

    So, you know, this is kind of a cool example. Um, but, you know, I, I, I wanna highlight how different the world is if software engineering is just this easy, you know, if you can just explain exactly what you want in plain English, um, and, and get that out.

23. 3:30

    Um, and so, you know, this is obviously kind of a toy use case and it, it's, it's perhaps useful, but we use Devin all the time ourselves when we're building Devin, actually.

24. 3:38

    Um, and by the way, I, obviously, I, I didn't make this website myself. I just said, "Hey, can you g- build me this website with the QR code and whatever?"

25. 3:45

    Uh, and Devin built that too. But, um, you know, here, here's a, here's a quick example of Devin, uh, that we're using ourselves in production. And so, you know, if you take a quick look here, for example, there's this whole search bar and, um, there's all the sessions, and you can search across sessions, right?

26. 4:00

    Uh, Devin actually made that in the Devin repository. Um, you can see here Bryce is on our team, and Bryce was asking, "Hey, Devin, can you go into the Devin sessions list, create a search bar component?

27. 4:09

    Here's what I need you to do." Um, and so there's a few features about this in particular that are obviously, um, tuned for working in a production codebase. You can see here that Devin started from a snapshot, so th- we have a machine instance loaded where it's cloned from.

28. 4:23

    It has a playbook, so it knows, like, a lot of the details about our repositories. And then it's also just able to generally work within our Git environment. So you'll see it just make a PR and interact with all those same tools.

29. 4:34

    And so I'll just kinda go through this quickly. So yeah, Devin says, "Absolutely." You know, makes the first pull request. Bryce continues, and again, you're, you're just giving feedback in plain English, right?

30. 4:42

    And you say, "Hey, this is a great start. You know, now could you, could you add a magnifying glass, um, and make it idiomatic? You know, use phosphor or luc- you know, it's up to you," right?

31. 4:51

    Uh, and Devin says, "Yeah, sure, I'll build that." Um, Bryce says, "Oh, by the way, no need to test, you know, I trust you." Um, mmm, and Devin says, "By the way, I'm dealing with a bit of an issue with the login process."

32. 5:01

    You know, it's just like you're working with, with another engineer, right? And Devin s- and Bryce says, uh, "Okay, bro." Uh, and you know, it kind of builds it all and gets the PR, and this PR was actually merged, and this is, you know, the search bar, right?

33. 5:13

    And similarly, you know, a lot of the API integrations that Devin has were built by Devin. You know, a lot of our own internal dashboards and metrics tracking within Devin were actually also built by Devin.

34. 5:24

    Um, and it's been kind of a, kind of a fun one to see, like, Devin building the company with the company as well. Um, so cool. Yeah. I, I wanna talk a little bit about, you know, our journey so far and, and about what's happening in the space as well.

35. 5:36

    And so, um, you know, we got started back in November, um, so it's been about seven months now. Um, it's kind of funny. We started in a hacker house in Burlingame, uh, and it was basically just a lot of us had already, like, lived together at that point.

36. 5:49

    You know, we'd all had our own journeys in AI, and we just knew that we wanted to build something together, and we obviously knew that we wanted to do something in code and build a coding agent.

37. 5:57

    Um, and then that hacker house in the [REDACTED:location]. After that, there was another hacker house in New York. Then there was another hacker house in the [REDACTED:location].

38. 6:03

    So we were actually-- we, we've been going back and forth between New York and the [REDACTED:location] for Basically the last seven months. I think at this point we are now gonna, like, settle in the [REDACTED:location], but it's been going back and forth and getting, like, a slightly bigger Airbnb each time because the team also gets a little

39. 6:15

    bit bigger. Um, but you know, why, why Devin in particular?

40. 6:20

    Um, and, you know, this is, this is a, um, particular question that I, that I'm really passionate about, which is, you know, language models have been pretty big. I, I think that's, that's fair to say.

41. 6:30

    Um, and you know, the first wave of generative AI is what I generally call these text completion products, right? And you know, that makes a lot of natural sense if you think about it, that obviously the interface of a language model is text completion, right?

42. 6:42

    You give it a prefix, and it completes the suffix from there. And so if you think about ChatGPT, if you think about a lot of these Q&A products, if you think about, you know, writing marketing copy or answering customer support or even GitHub Copilot and Cursor and, and products like that, you know, obviously very...

43. 6:59

    You know, a lot of these are really great products and, and, and very natural use case where you have the prefix so far and you're asking the models to complete what's next in the suffix, right?

44. 7:06

    And it does that for you, and that's a tool that's useful, right? And I think we're entering this new wave where, you know, we're going beyond that and actually introducing some amount of autonomous decision-making.

45. 7:17

    Um, and obviously, you know, that's typically referred to in our space as agents, right? And you know, there's, there's all sorts of new things that you unlock, right? There's a, there's a lot higher bar of consistency that you require, but there's new things that you unlock with that.

46. 7:29

    Um, and so it's, it's been an interesting one because it's, it's both a very deep core capabilities question of getting Devin to, to solve these, but also a pretty interesting product design problem because I think the, the UX of agents is, is something that's extremely new.

47. 7:43

    Um, and then why code in particular? You know, a few different things. Obviously, we're all coding nerds as well. You know, we're all engineers, and so the idea of teaching AI to code is, is, you know, one of the coolest things [chuckles] that we could think of.

48. 7:55

    But beyond that, I think there's a few particular reasons that code with agents works especially well. You know, one is that obviously there's so much more to being a software engineer than typing the code, right?

49. 8:06

    A, a lot of the work that you're gonna do is, you know, you're gonna be looking into a bug, you're gonna be looking at the different files of the codebase.

50. 8:13

    Maybe you're gonna be running this or that command. Maybe you're gonna be pulling up documentation. Maybe you're gonna run the front end yourself to, to reproduce the bug. You know, you look at this thing, you make it-- make this edit, you try it again.

51. 8:23

    Um, all, all of this work here obviously is... You know, that's what software engineering is, right? More so than just, uh, than just typing the code in the file, um, which leads very naturally to an agentic workflow.

52. 8:34

    You know, another part which I think is closely related is the ability to iterate with code feedback. Um, and so what I mean by that is, you know, if you were given an entire production codebase and you were told, "Hey, this has this one bug.

53. 8:46

    I need you to fix it. Here's the bug." Um, you know, and, and it's-- let's say it's, like, thousands of files and, you know, hundreds of thousands of lines of code.

54. 8:55

    I mean, it'd be pretty tough, honestly, for most humans. It's also gonna be quite tough for AIs as well. And obviously, the way that we do this in practice is, you know, you, you, you go and add print statements.

55. 9:06

    You pull up the logs. You check the monitoring. You know, you, you jump back and forth between different files. You try and diagnose it, right? Each of these things that you're doing, you know, you're, you're making a decision, and then you're running actual code to find out what happened.

56. 9:18

    And from that, you're, you're able to iterate, and it just gives you a much cleaner path to solve the problem in front of you. Um, and similarly, you know, that, that kind of lends very well to agents.

57. 9:26

    And the last thing I just wanna mention is, you know, how fast model agentic capabilities are improving. Um, and so, you know, two years ago, like, even something as simple as, as this name game demo, I think would've been almost unthinkable.

58. 9:39

    Um, and you know, you think about where things are going and where things are gonna be two years from now, I, I think there's a lot of, you know, the data, the, the, the right training and so on that's, that's really, really rapidly improving in the space.

59. 9:53

    Um, and then, you know, again, beyond the capabilities problem, there's actually a really deep UX problem as well. Uh, and at a high level, you know, I think what's kinda happening here is when we're building agents, and I think all of us in the space are quite new to agents, you know, the immediate first things I think

60. 10:08

    to map to are, you know, how we use software today and also how we talk with other humans, right? And so, you know, I mean, even a lot of the features in Devin are essentially looking over your own intern's shoulder.

61. 10:19

    You know, you can see their computer, and you can see what commands they're running and things like that. The thing is, I think an agent is actually pretty different from both.

62. 10:26

    You know, there's a lot of nuances and details of parallel work, information gathering, how it manages context, um, et cetera, et cetera, that are super, super different. And it's actually a quite deep problem from a product perspective as well.

63. 10:39

    And just to give you guys a bit of a sense of that, like, here's just a kind of, uh, a short list of some of the features that we've built into the product.

64. 10:46

    And so, you know, obviously there's Devin being able to use the shell, you know, edit code, browse the web, but there's all these other things, right? You know, being able to fork and roll back sessions.

65. 10:55

    You know, being able to handle integrations with Slack and GitHub. Being able to handle playbooks, to, to store machine snapshots, to keep track of secrets. You know, to be able to work with the right tools for verification.

66. 11:05

    You know, all of this is part of the actual product iteration, right? Which is, you know, on its own, I think already an incredibly, incredibly dense problem. And I think honestly, we're gonna see actually a lot more iteration with that over time, and I just wanted to show kind of, uh, a new feature which we just recently

67. 11:21

    shipped, which is the ability to use Devin's machine. Um, which is kind of-- a-again, it's, it's the kind of thing that's not always, uh... There, there's not necessarily a, a very close parallel in, uh, you know, in, in the software that we have today, right?

68. 11:35

    But the ability to just have a VS Code live share in Devin's machine and, you know, if you wanna collaborate with Devin and say, "Hey. Oh, there's this, these couple lines," like, you know, "You should make this edit.

69. 11:44

    I went ahead and did that edit for you," and you can just talk with Devin and do that, right? Um, so there's a lot more room to go and a lot to iterate on in the space.

70. 11:52

    Um- One of the thing, uh, of the things I wanted to mention too is just how much, you know, we've seen it changing our own workflow. Um, so you, you guys saw like a simple example of Devin building the search bar, but, you know, we actually handle tasks in a much more async way now.

71. 12:05

    Uh, one of the cool kind of features of Devin, I'd say, is, you know, if as an engineer you're, you're working on, let's say, four different tasks today, you know, you just give one to Devin number one, you give the second one to Devin number two, the third one to Devin number three.

72. 12:17

    You have four Devins that are all running in parallel, and it's kind of turning every engineer into an engineering manager is almost how I describe it. You know, I think the Devins are very, like, enthusiastic interns [laughs] is how...

73. 12:28

    What I'd say. I mean, they, they try very hard. You know, they're, they're... Obviously, they don't know everything. They get little things wrong. They ask a lot of questions, but, you know, you're kind of working with each of them and, and having them iterate.

74. 12:37

    And so here's just kind of a fun example. I mean, this is literally from, from earlier today. But, you know, we were talking about some particular feature and about what we wanted to build.

75. 12:46

    In this case, it was a pretty simple thing of changing the color, but it's just as simple as just saying in Slack, in the conversation, "Hey, [REDACTED:username], can you just change this thing?"

76. 12:53

    And then Devin goes and makes a PR, and then you hit merge, you know? And so, um, you know, we've had a lot of occasions where we're, you know, in the gym or in the car or something, and now you can actually write code 'cause, you know, you can tell Devin exactly what you want Devin to do.

77. 13:05

    Um, you just don't have your whole computer with you and can't type everything, but, you know, being able to just kind of describe what you want to Devin and then being able to review the code afterward, um, actually works really well.

78. 13:16

    So what's next? Um, you know, I, I, I think this is a really important question. Uh, and [laughs] obviously, I think the technology's extremely early today. Um, but, you know, where do these things go in a few years?

79. 13:26

    And also, what happens with software engineering? I think there's, there's been a lot of uncertainty about that question. And, you know, as we're using Devin more and more, I think one of the big things that we see actually is, this is kind of obvious perhaps, but Devin is not the one that decides what to do or what

80. 13:42

    to build, you know? And there's this core part of software engineering, the way I describe it is like software engineers everywhere, you know, are, are doing really two jobs at once, right?

81. 13:50

    And the first job is basically problem-solving with code. You know, you, you're, you're given a problem, and you're breaking down exactly what is the solution you're gonna build. You know, what is the architecture that you're gonna use?

82. 14:01

    What are all the flows and the details and the edge cases that might come up? And kind of architecting your exact solution. And then the second part is once you have that, you know, you're dealing with debugging or implementing different functions or writing unit tests or all of the other things that kind of go into this implementation

83. 14:18

    of something that you know you want to do, right? And, you know, I think right now the average software engineer is probably spending like 10 or 20% of the time on that first thinking part, and they're spending 80 or 90% of the time on that implementation part.

84. 14:30

    And, you know, what we really see is

85. 14:33

    Devin actually just frees you up to do more of the first part, you know? And I, I think the future of Devin, again, it's very, very early, but I think as Devin gets better, we're gonna see more of that, where Devin just frees up the implementation for you, where you don't have to go figure out how to

86. 14:45

    set up Kubernetes. You know, you don't have to go, like, debug all these, like, APIs that are broken. You know, you don't have to go, like, deal with version changes or migrations or all of these other things that, you know, take up a lot of time in software engineering, right?

87. 14:57

    But you actually are spending all your time on figuring out how to solve the problems in front of you. You know, it's, it's a little more like, um, a mix between, you know, a technical architect and a product manager almost, right?

88. 15:07

    And so, you know, I, I think software engineering, the job that we call software engineering, is gonna change. But I think practically, like, there's actually gonna be way more software engineers than ever, you know?

89. 15:17

    And, and I, I think there's a lot of precedent for that too, you know. Programming back then, you know, used to mean punch cards, and then after that, it used to mean assembly.

90. 15:26

    And, you know, and then after that, it used to mean C, right? And, you know, as these things have gone on, I mean, most people aren't using punch cards anymore.

91. 15:33

    But there's actually way more programmers than before, right? And, and I think one of the things that's, um, easy to underestimate is just how much more code there is to write.

92. 15:42

    Um, and you know, it's, it's, it's funny to think about, I think, because obviously we all love software here in this room. I would say I think software has been the number one driver of progress in the world in the last 40 or 50 years.

93. 15:52

    And yet despite that, I think, you know, our demand for software to be built is actually probably a lot more than 10X what we're currently getting. Um, and so, you know, I, I think what happens is we, we get to open up the power of software engineering to a lot more people, and every single software engineer gets

94. 16:08

    to be 5 or 10X more effective. But we actually do a lot more software engineering. Um, cool. Yeah, so, so that's all I had. Uh, but yeah, we'd love to open the floor if there's any questions. [audience applauds]

95. 16:25

    Yeah, right here in the front.

96. 16:27

    Oh, I'll shout out. Okay. Yeah. Well, thanks again. This is great. Uh, blew my mind, like, eight times. [laughs] So, um, how do we get access? Like, to me now [laughs]

97. 16:36

    Great question. Great question. So [laughs] so we've been ramping up access. Um, every, every week we've been letting on more and more people. We've also been sizing up with our enterprise customers.

98. 16:45

    We have a lot of wait lists to get through [laughs], so we're, we're, we're doing it as fast as we can. Um, but, but we, we'd love to get you guys access as soon as possible.

99. 16:52

    Thank you.

100. 16:53

     Yeah. Yeah. Um, all the way in the back over there. Yeah, in the red.

101. 16:59

     Hey. Uh, so I'm just wondering how much... Like, does Devin have access for running the code whilst in the back end? Like, how are you doing that analysis for correctness for the code in, like, the generation stage?

102. 17:09

     Yeah, exactly. So in our code base, for example, Devin has all the setup that it needs. It has a machine that's basically instantiated where it can run the dev environment, it can run the server, it can run the front end.

103. 17:18

     And so if it's... If you're asking it, "Hey, I need you to debug this particular thing," it'll just pull it up itself and then, you know, reproduce it, and then it'll debug it and try it again.

104. 17:26

     Yeah.

105. 17:26

     Cool. So it's, like, fully executable.

106. 17:28

     Exactly. Yeah. Yeah. Yeah. Any other questions? Right here.

107. 17:34

     So with all the simpler tasks potentially being solved by Devin software, how do we know if we forget about all the juniors, interns that don't know how to code yet? [laughs]

108. 17:48

     Yeah.

109. 17:49

     It would be good for you to repeat the question.

110. 17:51

     Oh, of course. Yeah. So, sorry. So, so someone asked, um, you know, with all of these simpler tasks getting solved, um, what happens to, you know, all the junior engineers or the interns who, who obviously need to learn how to code?

111. 18:02

     You know, I, I, I think- What happens honestly is I think that demand is going to just keep rising with supply, and I think the training process is gonna change a little bit.

112. 18:11

     But, you know, I think a lot of these core fundamentals of, you know, if you think of someone as when you say someone's a really great engineer, typically it do- you don't mean that they type really fast, although maybe they do that too, right?

113. 18:21

     You typically mean that they, they have a really great understanding of problems. You know, they know all the different architectures. They never miss an edge case, stuff like that, right?

114. 18:28

     And so those are the fundamentals that I think are always gonna matter, and I think, um, basically I think interns or junior engineers are going to get more exposed to getting to use those fundamentals earlier and earlier.

115. 18:42

     Yeah. Do you think one of the biggest open technical challenges you have to realize this vision of a future suite? Yeah. Okay. So someone asked what are, what are the, the biggest challenges to realizing the vision of a future suite?

116. 18:54

     You know, there's a lot. [laughs] Um, I mean, it's, it's basically everything as you can ima- I mean, there's speed, there's consistency, there's access, there's integrations, there's the right product UX.

117. 19:07

     Um, you know, and I think all of these things ... W- one of the cool things I think is just how much of a rising tide there is everywhere.

118. 19:13

     And so, you know, o- obviously we're, we're gonna do our best work on it, but, you know, every, every new hardware release is, is amazing for it. You know, e- every, every new foundation model that comes out is amazing, you know, every new piece of agentic research.

119. 19:25

     And I think this is the kind of thing where, um, I think there will be a lot of different optimizations that come in different parts of the stack that make this agentic flow better and better and better.

120. 19:35

     Um, but, uh, yeah, it, it won't just be one small thing. But I, I think it'll be, it'll be pretty fast.

121. 19:41

     And that's all the time we had, so thank you so much.

122. 19:44

     Thank you guys so much. [outro music]