#define AI Engineer

Talk transcript

1. 0:00

   [upbeat music] [applause] Thank you.

2. 0:21

   Thank you. [applause] Well, hello, hello. Is, uh, mic working for you? Yeah?

3. 0:30

   Check, check, check. One, two, three.

4. 0:31

   Okay. Uh-

5. 0:32

   All right, first hard technology problem of the day down.

6. 0:34

   Yeah, yeah. Well, the Wi-Fi is the other one. [laughs] [laughs]

7. 0:38

   Um [laughs], everyone here knows. Um, so Greg, welcome to AI Engineer. Thank you so much for taking the time.

8. 0:43

   Thank you for having me. [laughs]

9. 0:44

   Um, we're gonna go a little bit chronologically, and, uh, a lot of people sending questions, and I've sort of grouped them up for you. So we'll just get right into it.

10. 0:53

    Uh, so, you know, you k- you know I did some deep research on you. Uh, you started out-

11. 0:58

    You literally did peep research

12. 0:58

    ... with, with deep research. Um, I called it peep research because we're researching a person. Uh, you actually did theater growing up, and chemistry and math, and you wrote a calendar sched- scheduling app, and that's what got you into coding.

13. 1:10

    But, like, what really inspired your love for coding? Like, why, why are you the coding guy?

14. 1:15

    Well, the funny thing is I thought I was going to be a mathematician when I grew up.

15. 1:19

    Yeah.

16. 1:19

    You know, I'd read about people like Galois and Gauss, you know, who were working on these, like, 100, 200, 300 year time horizons, and I was like, "That's what I wanna do."

17. 1:28

    If anything that I come up with is ever used while I'm still alive, it wasn't long term enough. It wasn't abstract enough. Um, and I was writing this chemistry textbook after high school, sent it to one of my friends who'd done something similar in math, and he said, "No one is gonna publish this.

18. 1:42

    You can either self-publish." I was like, "Oh, sounds like a lot of work, a lot of capital." "Or you could make a website."

19. 1:48

    Mm-hmm.

20. 1:48

    And I was like, "Guess I'm gonna learn how to make a website." And so I literally went on W3Schools and did their PHP tutorial. How many people here remember W3Schools?

21. 1:58

    Yeah, a decent number of hands. Um, and I remember the very first thing I built was a table sorting widget, right? I had this picture in my head of what it would be.

22. 2:07

    And I remember the moment that I clicked the column, and it sorted according to that column, which was exactly the thing that I wanted, and I was like, "That was magic," right?

23. 2:14

    And I was like, "This is so cool." 'Cause the thing about math is that you think hard about a problem, you understand it, you write it down in an obscure way you call a proof, and then, like, three people will ever care.

24. 2:25

    Mm-hmm.

25. 2:26

    Right? [laughs] But in programming, you write it down in an obscure way we call a program, and then maybe only three people ever read that program and care about the code, but everyone gets the benefit.

26. 2:37

    No one has to understand the details. That thing that was in your head, it's real. It's in the world. And I was like, "That, that's the thing I wanna do.

27. 2:43

    Forget about that 100-year time horizon. I just wanna build." [laughs]

28. 2:47

    Uh, you do just wanna build. Uh, it, it's, uh... So you were so good at it that somehow, somewhere, you got cold emailed by Sc- Stripe while you were still in college.

29. 2:56

    That's right.

30. 2:57

    Uh, what's the story of how... First of all, how did they find you, and what was it that convinced you to drop out to join them?

31. 3:01

    Well, so I had mutual friends with all the people at, at Stripe, the, you know, giant company of, like, three people at the time. Uh, [laughs] and, uh, uh, th- they, they'd asked, you know, it was the usual thing where they'd ask someone at Harvard who the, you know, people around campus had talked to, uh, who they might

32. 3:17

    recruit, where my name came up. They asked the same for the people at, at MIT because I actually had dropped... I'd, I'd been at Harvard and actually dropped out to go to MIT.

33. 3:24

    So I, I had the advantage of, uh, I guess, you, you know, uh, get- getting upvotes on both sides. Um, but I remember when I met the, Patrick, and it was...

34. 3:35

    You know, I'd just flown in. It was, like, late at night. The, you know, it was storming. And, uh, I, I showed up, and we just started talking about code, right?

35. 3:42

    And it was just like one of those moments where you're like, "This, this is the kind of person that, uh, that I've wanted to work with and been looking for."

36. 3:47

    Uh, and so I ended up dropping out of MIT, uh, and, uh, you know, flew out and been out here ever since.

37. 3:53

    Yeah, yeah. Uh, we have a special... We have some guest questions sprinkled along the way, as you know. Uh, so a guest question from someone named Matthew Brockman. [laughs]

38. 4:01

    I've heard of him.

39. 4:01

    CTO of, uh, Julius AI. When do you think our parents will give up on the dream of you finishing your degree? Maybe- [laughs]

40. 4:08

    Maybe Harvard or UND will take you back.

41. 4:10

    Yes. Uh, well, ne- ne- never. Um- [laughs] It was definitely... I've, you know, I think it was, no matter where you're going, if you tell your parents you're leaving Harvard, it's gonna be hard.

42. 4:19

    Yeah.

43. 4:19

    Um, you tell your parents you are leaving school altogether, it's going to be difficult. Um, and I think that, you know, it was actually, um, due, to their credit, you know, I think even though it was difficult, um, they, they were like, "That, you know, we, we trust you, like, you, you must see something and, and understand

44. 4:32

    something from, from where you sit that's hard for us to see from, from halfway across the country." Um, but yeah, I think that, that as, you know, did Stripe and, uh, and had a good time and, and actually learned things, um, and, uh, it turned out it was a real company and not just, uh, uh, you know,

45. 4:48

    just dropping out and doing nothing, uh, I think that, that they, they really were, were, uh, you know, have, have warmed up to it. And so, um... [laughs] [laughs]

46. 4:55

    Uh, I think they're very proud of you.

47. 4:57

    Yes, absolutely.

48. 4:58

    Um, so you, you were with Stripe from four to 250 people as the first CTO, eventually. Um, one other thing I, I found recently that Hacker News maybe doesn't know is apparently the call us at installation only happened, like, a handful of times.

49. 5:09

    It wasn't, like, a thing at Stripe. Was that-

50. 5:12

    That, that's, I think that's true. Um, yeah. It is-

51. 5:15

    Yeah

52. 5:15

    ... it is the thing that, that, you know, it's like survived the, uh, the, the lore.

53. 5:19

    It's a, it's an urban legend because it's, like, so cool. It's like you get so, so customer obsessed. Anyway, so what else do people get wrong about early Stripe?

54. 5:25

    Like, what do we wanna clear the air of?

55. 5:26

    Yeah. Well, I think people don't understand how hard it was, right? It was just like, um, like I remember, um, you know, first of all, the, the kind of thing that we did a lot of is that we added all of our customers on G Chat, and so it was very much the case that we were in

56. 5:40

    constant contact with them. And so even if you're not literally sitting over their sh- their shoulder, you're doing the, the next best thing. Um, but I remember, um, like one, I, you know, one, one, one day we realized that I, you know, the, the, the payment backend that we were on, it just wasn't going to scale.

57. 5:55

    Uh, we absolutely needed to be on Wells Fargo, and we got sort of the deal done, but now we need to do a technical integration, and they said, "Well, this technical integration is going to take, like, nine months because that's how long it takes."

58. 6:07

    And we were like, "That's crazy. Like, you're a startup. Like, we can't sit around waiting nine months to get this thing done." Um- And so actually, in 24 hours, uh, we completed it, uh, by just basically treating it like a college problem set.

59. 6:19

    Uh, and it was, you know, I, I was implementing everything. John was working from the top of this test script and testing everything and being like, "This is broken."

60. 6:26

    Dara was starting from the bottom and working his way up. And, uh, in the morning, we got on with, uh, with the, uh, certifying, uh, person, and we sent some, some test messages and there was an error.

61. 6:37

    And the person's like, "All right, I'll see you next week." Um, 'cause that's how all their customers operate, right? There's an error, like, you know, clear you need to send it to your dev team.

62. 6:43

    And we were like, "No, no, no. There must just be like, like some sort of glitch in the system." Like, and we were just... Patrick was just, like, talking to keep her on the line, and frantically, like, I was there editing the code. [laughs]

63. 6:53

    And so we got, like, five turns in, uh, and we actually failed. Uh, but fortunately, she, she was nice enough to reschedule two, two hours later. Uh, and there, then we passed.

64. 7:01

    And so you realize that was, like, six weeks worth of normal dev work that you got done in that moment because you didn't just accept the, like, arbitrary constraints of how other organizations would work.

65. 7:10

    Yeah. Yeah. Do you... I think there's a... Do you think there's a lot more opportunity like that in most jobs? Like, how do you, how do you advise other people to be that, I guess, fast or, like, to cut that many cycles?

66. 7:22

    Yes. I mean, I think that... I- the way I think about it is that if you think from first principles, you can find where things need to be slow or done the way that they're normally done or whatever those things are.

67. 7:34

    Those exist, right? The general principle of, ah, just don't worry about the constraints and just do the thing, um, I think that, that, that is not 100% true. I think it's really about mapping to: Where is there unnecessary overhead that's there for constraints that are no longer applicable, that, that don't apply, uh, to your specific circumstance?

68. 7:51

    And I think this is especially true in this world that we're in now with AI that's ac- accelerating productivity so much.

69. 7:56

    Yeah. Just fire off a Codex. Why not, right?

70. 7:58

    Yeah.

71. 7:59

    Um, o- one thing I, one thing I... One last thing about your sort of pre-OpenAI life was, uh, independent study. I just, I f- I found that just it's a recurring theme from high school.

72. 8:08

    Did y- you did Recurse Center?

73. 8:09

    I did.

74. 8:10

    Um, and your sabbatical as well. Just so you've just done it repeatedly. What makes independent study effective? Like, I think there's a lot of people who don't do a good job of it and kind of waste a year.

75. 8:21

    What, what, what do you do that makes it so effective?

76. 8:23

    Well, I think it was a key part of how I grew up. Um, you know, in f- in, uh, in sixth grade, my dad taught me algebra, and in seventh grade, showed up at the high school.

77. 8:34

    It's the first time that y- you track into advanced math, pre-algebra, and we went to the teacher, we're like, "Can he skip, uh, this and go directly to the, the eighth year, the eighth grade course?"

78. 8:43

    And the teacher looked at my mom and me very condescendingly and was like, "Every parent believes that their child is special." Um. [laughs]

79. 8:51

    And after, like, a month of being in this teacher's class and, you know, I was paying no attention and just doing, you know, calculator games in, in the back, and she'd try to trip me up and, you know, call on me to answer questions from the whiteboard, and I would just get them all right.

80. 9:02

    She was like, "All right," like, "fair enough. Uh, your, your child should be, uh, in the next year." Um, it- but then when I was in eighth grade, there was no more math left in my middle school.

81. 9:11

    I didn't have a car, so I had to do online courses. And in that one year, I ended up doing three years' worth of high school math.

82. 9:18

    Yeah.

83. 9:18

    And so I think that a, for me, a lot of it is about suddenly these... If you're, if you're excited about something independently, it's something you wanna do, then you can break the constraints there as well.

84. 9:27

    Uh, you can do three years of math in one year. And then it compounds, because the next year I was at my high school, finished math there, and then all through 10th, 11th, and 12th grade, I, I had, you know, no more math.

85. 9:39

    So I did have a car, and I was able to go to the University of North Dakota, take whatever classes I wanted there. And so I think that, that, that kind of compounded, compounded, compounded to learning programming.

86. 9:49

    And then I think that, that the way I learned programming, it's very much self-study, just building things and, and experiencing things out in the world. And so I think that the thing I would just advise is, like, if you have an opportunity to explore and you have a passion, you're actually enjoying it, just go deep, right?

87. 10:06

    And by the way, it's not always fun, right? I think that it is very easy to, uh, get kind of, y- you know, sort of feel like, ah, I got kind of bored.

88. 10:14

    But i- if you just push through those hurdles, then I think that the, that the reward is worth it.

89. 10:17

    Yeah. You self-studied machine learning, too.

90. 10:19

    That's right.

91. 10:19

    Like, that was a whole period of your life. Um, any particular highlights from there? It s- it sounds like you talked to Geoff Hinton at one time.

92. 10:26

    I did talk to Geoff Hinton. [laughs]

93. 10:27

    Yeah.

94. 10:27

    Yes.

95. 10:28

    And, like, was... You know, w- did, did that help? Or what was the most helpful thing? Like, you-

96. 10:32

    Well-

97. 10:32

    ... became a machine learning practitioner.

98. 10:33

    Well, so, so when I, when I started out... So, you know, I'd been, I'd been at Stripe. I was reading Hacker News posts about deep learning and- [laughs] Yeah, it was like, you know, there's a deep learning for acts, like, every day it felt like.

99. 10:45

    And this was, you know, 2013, 2014, and I was like, "What is deep learning?"

100. 10:50

     Mm-hmm.

101. 10:50

     And I knew, like, one person in the field, and so I talked to them. They introduced me to some more people, and then they introduced me to more people.

102. 10:56

     And the thing that surprised me was I kept getting introduced to a bunch of my smartest friends from college. And I was like, "That's interesting. All of these people ended up in this field?

103. 11:04

     Like, what's going on?" And I started to realize that, that there was something real that was building, right, that was being developed, that people were really making these systems do material, new things that computers were not able to b- do before.

104. 11:18

     And I was like, "That, that is the thing." Um, and so after I left Stripe, you know, I knew I wanted to do something in AI, um, start an AI company, but I didn't really know how to contribute, what my skills would be useful for.

105. 11:31

     And, uh, so I was in New York and I was like, "You know what? I'll build a GPU rig and see if I can do some Kaggle competitions." And so I went on Newegg and just, like, you know, bought some, uh, some Titan X cards and, uh, it was really cool, you know, physically assembling this machine.

106. 11:45

     And, uh, you can find some, some tweet from, from 2015. When I powered it on, you see all this, like, green and all the fans going.

107. 11:51

     Yep.

108. 11:52

     And I was like, "This, this is what computers are meant to be." [laughs]

109. 11:56

     Uh, I think, uh, many f- folks in the audience have that- [laughs] ... exact experience as well. Um, awesome. Okay. So what convinced you that AGI was possible? Like, you, you had a point where you were sort of disillusioned with it.

110. 12:07

     You wrote, you tried to write a chatbot. You didn't... It, it didn't work. But what made you go all in on it?

111. 12:12

     Yeah. Well, so, you know, part of, part of the journey for me was reading Alan Turing's 1950 paper, "Computing Machinery and Intelligence." This is the Turing test paper. How many people have, have read it?

112. 12:23

     You-- fewer hands than, than W3Schools. [laughs] Uh, but equally as important. Uh, worth reading. Uh, the thing that is so fascinating to me is he lays out in the beginning, okay, Turing test, it...

113. 12:34

     this idea of just does a machine think? Is it intelligent? And you can say it's intelligent if, you know, a human can't tell the difference between talking to it and talking to a human, fine.

114. 12:43

     But the thing that was-- that has not really become as embedded in the pop culture but to me was so astounding, was he said, "Well, how are you going to program an answer to this?

115. 12:53

     You will never be able to write down all the rules."

116. 12:55

     Mm-hmm.

117. 12:55

     But what if you could build a child machine that learns like a human child, and then you just apply rewards and punishments and boom, it's going to, uh, it's going to, to be able to, to pass the test.

118. 13:06

     And I was like, that, that is the kind of technology that we have to build because as a programmer, you have to understand everything. You have to understand the rules of how to solve the problem.

119. 13:15

     But what if the machine can understand things and solve problems that you yourself cannot understand? Like, that feels fundamental, right? That feels like how you actually solve problems that are important to humanity.

120. 13:26

     And I-- this was, you know, 20- 2008 or so that I read this, and I went to my professor and, uh, who's an NLP professor, and I asked if I could do some research with him.

121. 13:37

     And he said, "I... Yeah, here are some parse trees." And I was like, "Okay, this is not what Turing was talking about."

122. 13:42

     Yeah.

123. 13:42

     Um-

124. 13:42

     This is like WordNets and-

125. 13:44

     The whole thing.

126. 13:44

     Yeah.

127. 13:45

     Exactly. So it's like, you know, definitely, definitely a little bit of trough of sorrow there. Um, but with deep learning, the thing about deep learning that's magic is that, you know, it really started in, to show, show promising results 2012 with, with AlexNet, right?

128. 13:58

     And, and that it just blew everyone out of the water in the ImageNet competition. And so suddenly you have this, like, general learning m-machine. You know, it's got a little bit of a prior in there of, of, of convolutions.

129. 14:10

     But it's better than forty years' worth of computer vision research, right? People trying to write down all the rules as well as possible. And then people are like, "Well, okay, it works in vision, but it's never gonna work in my field.

130. 14:21

     It's never gonna work in machine translation, never gonna work in, uh, in, you know, in NLP, never gonna work in this or that." And suddenly it starts being the best in all those areas.

131. 14:31

     Suddenly the walls between these departments are being torn down. And you're like, that, that is what Turing was talking about. And so I think for me, just seeing the, the type signature of this technology, and by the way, this technology's not new, right?

132. 14:44

     Neural nets were really... Like, if you go back and read the, uh, the McCulloch-Pitts, uh, neuron paper from, like, 1943 or so, um-

133. 14:52

     I told people, I told them to give homework to people, so-

134. 14:54

     Okay. Yeah, there you go.

135. 14:55

     Look that up. Yeah.

136. 14:55

     Yes. [laughs] Class is assigned. Um, the, there, the i- the images in there, they look just like the kinds of images that you see now-

137. 15:03

     Yeah

138. 15:03

     ... of just, like, you know, layers of neurons and things like that. And so you just realize there's something deeply fundamental about what we're doing. And, uh, you can find these, these, uh, you can find this paper, um, from 1990-- the 1990s talking about what caused the deep learning winters and that it was these neural net people,

139. 15:19

     they have no new ideas. They just wanna build bigger computers.

140. 15:23

     Yeah.

141. 15:23

     And I'm like, "Yes, [laughs] that's what we need to do." Um, and so I think that all of this together just feels like we are, we are to some extent continuing this wave, this seventy-year history, um, and in, in many ways, um, you know, the whole computing industry has been really trying to build up to the point that

142. 15:40

     you have machines that are able to perform the kinds of tasks that we're just starting to scratch the surface to solve new problems that humans cannot, to be, be assistive to us in our daily lives, to not have to, you know, be typing with our, with our, you know, meat sticks, but instead to have something that you

143. 15:55

     can interact with just like a person, where the machine comes much closer to you rather than you closer to it and having to learn assembly language or, you know, whatever it is.

144. 16:03

     Um, and so to me it felt like all of the factors were lined up and now we just need to build.

145. 16:08

     Yeah. Um, I, I like that consistent theme that you keep coming back to, we just need to build. Um, so in 2022 you wrote that it's time to be an ML engineer.

146. 16:17

     Actually, I have a personal friend, uh, who was in-- read that post and cold emailed you and joined OpenAI and all that. Um, you said that great engineers are able to contribute at the same level as great researchers to future progress.

147. 16:27

     Is that, uh, is that still true today? You know, I think a lot of engineers look at the researchers who are making millions of dollars and they're like, "How do I contribute as much?"

148. 16:37

     You know.

149. 16:37

     I, I think it's absolutely, if not even more true. Um, I think that, like, if you look at the phases of deep learning research since 2012, I think at the beginning it really was, um, and this is kind of what I expected when we started OpenAI, you know, just like research scientists who had gotten a PhD who

150. 16:53

     would go and kind of come up with ideas and test them out. And you know, there's, there's engineering to be done. If you actually look at AlexNet itself, you know, it's fundamentally the engineering of let's get fast convolutional kernels on a GPU.

151. 17:05

     Um, and, and, uh, fun, fun fact is people who were in the, the lab with Alex Krizhevsky at the time, uh, were-- actually felt very bad for him because they were like, he has some fast conv kernels for, uh, uh, for, you know, some, some image dataset that, uh, uh, doesn't really matter.

152. 17:20

     But you know, Ilya was like, "Well, clearly we just need to apply this to ImageNet. It's gonna be great." Right? So it's like the combination of great engineering together with the idea of what to do with it, right?

153. 17:30

     That that's what, what makes the magic work. Um, and uh, uh, the thing that I think is still true and even more true is, okay, so the engineering required, it's now not just let's build some kernels, but let's build a system, or let's like actually scale to hundred thousand GPUs.

154. 17:49

     Let's actually, you know, sort of do this crazy RL system that orchestrates things in all sorts of ways. Um, so the idea, if you don't have the idea, you're dead in the water.

155. 17:57

     There's nothing to do.

156. 17:58

     Yeah.

157. 17:58

     But if you don't have the engineering, that idea is not gonna, it's not gonna live and see the light of day. And so you need to have both of these coming together harmoniously.

158. 18:06

     Yeah. I think the Ilya-Alex relationship is really em-emblematic of like the research engineering, uh, partnership that now is the philosophy at OpenAI.

159. 18:15

     That's right. Yeah, and if you look at how OpenAI operates, like I think from the very beginning we had this ethos of engineering and research be valued, um, and, and work together, um, as partners.

160. 18:25

     Yeah.

161. 18:25

     And I think that that is something that we, you know, it's like something that we, we really work at every day.

162. 18:30

     Yeah. Uh, it's my explicit goal to try to throw- Uh, curveballs [laughs] in this, in this stuff. So, uh, in terms of the relationship between engineering and research, what did OpenAI do wrong in the early days that you do well now?

163. 18:43

     Um, well, I think that the relationship between engineering and research, the way I think about it is you never fully solve it, right? You just sort of solve the current level of problem, and then you move on to the next level of sophistication.

164. 18:55

     And I noticed that actually, the kinds of problems that we ran into were basically the same problems that had been run into at every other lab, and it was just like, you know, either we would be further along or the, the, there'd be a slightly different variant of it.

165. 19:05

     And so I think there's something deeply fundamental about this. Um, so the, the ver- at the very beginning, I could really see people who came from the engineering world, people who came from the research world, just sort of thinking about system constraints very differently.

166. 19:18

     And so as an engineer, you're like, "Hey, if I've got an interface, you should not care what's behind that interface. We agreed on the interface. I can implement it however I want."

167. 19:25

     Whereas if you're a researcher, you're like, "If there's a bug anywhere in the system, all I'm gonna get is just slightly degraded performance. Not gonna get an exception. Not gonna get indications of where, and so I am responsible for understanding everything."

168. 19:37

     The interfaces, they don't matter. Unless they're like truly rock solid, and I can just like never think about it, which is a pretty high bar-

169. 19:44

     Yeah

170. 19:44

     ... um, then I am actually responsible for, for this code, and that causes friction, right? Because then how do you actually work together? And I saw a project very early on where the, you know, the, the people from the engineering background would write the code, and then there'd be this big debate over every single line.

171. 19:59

     And I was just like, "This is never gonna move. It's gonna be so slow." And instead, the way that we ended up proceeding was, um, so I actually worked on that directly, and I'd come up with like five ideas at a time.

172. 20:10

     Someone from the research side would say, "These four are bad." I'd be like, "Great. That's all I wanted." Right? And so the value that I think we've really realized is critical and that I tell people from, from the engineering world coming into OpenAI, um, is technical humility, right?

173. 20:24

     It's like you're coming in because you have skills that are important, but it's a totally different environment from, you know, something like a traditional web startup. And figuring out when those intuitions apply and figuring out like when to leave them at the door is super hard.

174. 20:39

     And so the most important thing is to like come in, really, really listen and kind of assume that, that, that there's something that you're missing until you deeply understand the why.

175. 20:47

     And then at that point, great. Make the change, like change the, the, the architecture, change the abstractions. Um, but I think that that kind of approach of just really, really read and listen and understand with that humility, um, that that is, I think, a really key determiner.

176. 21:03

     Yep. Awesome. Um, we're gonna tell some stories from recent launches of OpenAI, the greatest hits. Uh, so one of the things that, uh, is, is kind of interesting is just scaling in general.

177. 21:13

     Uh, everything breaks at different orders of magnitude. So in- when ChatGPT launched, you got a million users in five days. This year, when 4.0 Image Gen launched, you got 100 million users in five days.

178. 21:24

     How do those two periods compare?

179. 21:26

     Uh, they echo very similarly in a lot of ways. You know, the thing about ChatGPT, uh, it was supposed to be a le- low-key research preview.

180. 21:35

     Mm-hmm.

181. 21:36

     And we put it out very, you know, sort of chilly, and then suddenly everything was down. And we, you know, we kind of anticipated that ChatGPT would be a very popular thing, but we thought that GPT-4 would be necessary to get it there.

182. 21:51

     Had it internally as well. So you just weren't impressed by people-

183. 21:53

     Exactly, right? It's like you... That's the other thing about this field, is you update so quickly.

184. 21:57

     Yes.

185. 21:57

     Right? It's like you see magic, and you're like, "This is the most amazing thing I've ever seen." And then you're like, "Well, why can't it like, you know, why, why can't it like merge-

186. 22:04

     Write jokes

187. 22:04

     ... 10 PRs for me?"

188. 22:05

     Yeah.

189. 22:05

     Exactly. Um, and the Image Gen moment was very similar in terms of it was just so, so loved and so popular, and it just went viral in, in ways that, uh, you know, just like the numbers were just o- off the charts.

190. 22:21

     And so internally, we actually did something that we really, really try not to do, um, which is we pulled a bunch of compute from research for both of these launches, actually.

191. 22:28

     Ah.

192. 22:29

     Um, because that's mortgaging the future, um, to make, make, make the system work. Um, but if you can actually deliver and keep up with demand, then of course people get to experience the magic.

193. 22:38

     And I think that, um, that, that's something that is really worthwhile, and, and it's really important to sort of, you know, maximize those moments. Um, so I think that, that, that we really have that same ethos of really serving the user, really trying to push for the technology and just do things that are materially new, that no

194. 22:55

     one's ever seen before. Um, and then whatever it takes to get those out into the world to make those successful, that that's what we do.

195. 23:01

     Amazing. Um, well, I mean, incredible job. Um, GPT-4 launch. So I'm told your wife drew the, uh, joke website.

196. 23:09

     That's true. Yeah. Fun-

197. 23:11

     In the audience

198. 23:11

     ... fun, fun Easter egg. [laughs] My handwriting was so bad, uh, that even our AI couldn't tell what to do with it.

199. 23:16

     Um, so like, uh, apparently... Did you improvise some of this? I, I, I heard.

200. 23:21

     I, I-

201. 23:22

     Through the grapevine.

202. 23:23

     Yeah, definitely. Definitely. Like, you know, usually when I, when I do these kinds of demos, like I've tested the general shape of them ahead of time. Uh, but I've always had a...

203. 23:30

     Like it's very easy in this field to have ones that are just like if you slightly typo a character or something, then the demo will not work.

204. 23:36

     Yeah.

205. 23:36

     Um, I don't like doing those. I like to have some robustness to it. So there's always variation in terms of, of what actually ends up get- being shown.

206. 23:42

     To me, this was the first time I think the world ever saw vibe coding. Um, it is now a thing. What are your thoughts on vibe coding?

207. 23:50

     Uh, well, I think that vibe coding is amazing as an empowerment mechanism, right? And I think it's sort of a representation of what is to come. And I think that the specifics of what vibe coding is, I think that's gonna change over time, right?

208. 24:05

     I think that you look at even things like Codex, like to some extent, I think our vision is that as you start to have agents that really work, that you can have not just one copy, not just 10 copies, but you can have 100 or 1,000 or 10,000 or 100,000 of these things running.

209. 24:22

     You're gonna want to treat them much more like a coworker, right? That you're gonna want them off in the cloud doing stuff, being able to hook to, hook up to all sorts of things.

210. 24:28

     You're asleep, your laptop's closed, it should still be working. Um, I think that the, the, the, you know, current conception of, of vibe coding in an interactive loop, um, you know, that that's something that I, I think is like...

211. 24:40

     You know, it's, it's... I, okay, so my, my prediction of what will happen is, like, I think there's going to be more and more of that happening, but I think that the agentic stuff is gonna also really intercept and overtake.

212. 24:48

     And I think that all of this is just going to result in just way more systems being built.

213. 24:54

     Um, and the thing that, that I think is also very interesting is that a lot of the vibe coding kind of demos and, and the cool, the cool flashy stuff, um, for example, make- making the joke website, it's making an app from scratch.

214. 25:06

     But the thing that I think will really be new and transformative and is starting to really happen is being able to transform existing applications to go deeper, um, and that be able to...

215. 25:17

     You know, like, I think so many companies are sitting on legacy code bases and doing migrations and updating libraries and changing your COBOL language to something else is so hard, and it's actually just not very fun for humans.

216. 25:30

     And, uh, I think we're starting to get AI that are able to really tackle those problems. And so the thing that I love about where vibe coding started has really been, like, with the most, like, just, like, make cool apps kind of thing, and it's starting to become much more, like, serious software engineering, and I think that

217. 25:44

     going even deeper to just, like, making it possible to just move so much faster as a company, um, that's, I think, where, where, where we're headed.

218. 25:53

     Yep. Uh, speaking of Codex, uh, I've heard that you've just, it's kind of your baby a little bit. Um, and y- you've started... You, I think on the live stream you were talking a lot about just make things modular and well documented and all, all that good stuff.

219. 26:05

     Like, how do you think Codex changes the way that we code?

220. 26:10

     Um, well, I definitely think that, that it's an overstatement to say it's, it's my baby. Like, I think that there's, um, a really incredible team.

221. 26:16

     Yeah.

222. 26:16

     Um, and, and, uh, that, you know, I've, I've been trying to support them and, and, and their vision and, um... But I think that, that the direction is something that is, like, just so, um, so compelling and incredible to me.

223. 26:29

     Um, the way that, that, uh... And, and sorry, could you repeat the, the-

224. 26:32

     How, how does Codex change the way, the way that we code?

225. 26:35

     I see. Yeah. The thing that has been most interesting to see has been when you, uh, realize that the way you structure your code base determines how much you can get out of Codex, right?

226. 26:46

     The, the... If you match the strength of, like, basically all of our existing code bases are kind of matched to the strengths of humans, but if you match instead to the strengths of the models, which are sort of very lopsided, right?

227. 26:56

     Models are able to handle way more, like, diversity of stuff, but also are na- not able to, like, sort of necessarily connect deep ideas as much as humans are right now.

228. 27:06

     And so what you kinda wanna do is make smaller modules that are well tested, that have tests that can be run very quickly, um, and then fill in the details.

229. 27:17

     The model will just do that, right? And it'll run the test itself. And the connections between these different components, kind of the architecture diagram, like, that's actually pretty easy to do, and then it's the, like, filling out all the details that is often very difficult.

230. 27:30

     And if you, if you actually do that, you know, what I described also sounds a lot like good eng- software engineering practice. Um, but it's just, like, sometimes because humans are, are capable of holding more of this, like, conceptual abstraction in our head, we just don't do it, right?

231. 27:44

     That, like, yeah, it's like, you know, it's a lot of work to write these tests and to, you know, to flesh them out. And, uh, the, you know, the model's gonna run, like, these tests, like, 100 times or 1,000 times more than you will, and so it's gonna care, like, way, way more.

232. 27:56

     So in some ways, the, the direction we wanna go is build our code bases for more junior developers, um, in order to actually get the most out of these models.

233. 28:06

     Um, now it'll be very interesting to see as we increase the model capability, does this particular way of structuring code bases remain constant? And I kind of think that it's a pretty good idea because, again, it starts to match what you should be doing for, for maintainability for humans.

234. 28:23

     Um, but yeah, I think that to me that the really sort of exciting thing to think about for the future of software engineering is what of our practices that we've kind of just cut corners for do we actually really need to bring back in order to get the most out of our systems?

235. 28:38

     Yeah. Um, can you put numbers on, like, ballpark numbers on the amount of productivity you guys are seeing with Codex internally?

236. 28:45

     Um, I, yeah, I, I don't know what the latest numbers are. I mean, there's definitely double-digit percent, uh, of our, of our PRs are written, uh, low, low double digit, um, written entirely by Codex.

237. 28:56

     Um, and that's super cool to see. Um, but it's also, like, I, you know, that it's not the only system that we use internally, and I think that, um, to me it's, it's still in the very, very early days.

238. 29:07

     Um, it's been exciting to see some of the external metrics. Um, like I think we had 24,000, uh, PRs that were merged in like the last day, uh, in, in public GitHub repositories.

239. 29:17

     And so it's just like, yeah, this stuff is all just getting started.

240. 29:20

     Yeah, it's doing a lot of work. Uh, guest question from Dylan Patel on scaling and, uh, reliability. Um, so as... We're, we're doing more tasks that take longer and utilize more GPUs.

241. 29:32

     They're also just unreliable. They fail a lot, right? And, and this is just well known. Um, so this causes training to fail as well. So like, h- h- but like, you know, you've, you've mentioned that you can sort of just restart a run and that's okay.

242. 29:43

     Like, how do you deal with this when you have to train long-horizon agents, right? Because you can't really restart something that has a trajectory that's kind of halfway, that is maybe non-deterministic.

243. 29:52

     Yeah. I mean, I think that there's a bunch of problems that you kind of solve, and then you make the models more capable, and then you have to resolve them.

244. 30:01

     And so yeah, when the, the rollouts are short, you know, 30 seconds, you kind of don't care that much about this problem. If they're going to be days, now you really care about this problem.

245. 30:12

     Yep.

246. 30:12

     And you have to start thinking about how to snapshot state and a bunch of things like that. Um, the short answer is that I think that there's a, this like ladder of complexity that you keep climbing with these training systems, and it goes from, you know, like a couple of years ago, all that we cared about was

247. 30:27

     just doing good old-fashioned free training, right? And that's like very ch- checkpointable. Um, and even there it's not trivial, right? It's like, you know, if you go from checkpointing once in a while to, like, you want to checkpoint every single step, now you need to think really hard about, uh, about how you're going to avoid copies and

248. 30:41

     blocking and all these things. Um- Then for something like these more complicated RL systems, there's still checkpoint ability in terms of, you know, maybe you care about, like, uh, you know, checkpointing your cache, so you don't have to recompute everything.

249. 30:53

     Um, and the nice thing about our systems is that, you know, language models are-- their state is very explicit, right? And it's something that actually can be stored, um, something that you actually can, can handle.

250. 31:03

     Whereas if you have tools that you're hooked up to that are themselves stateful, maybe those are not something you can restart and recover from. And so I think that, that if you consider the whole system end to end, thinking about what checkpoint ability looks like.

251. 31:17

     And there's also a question of maybe it just doesn't matter, right? Maybe it's fine that you restart the system, and you get some little wiggle in your graph. But these models are smart-

252. 31:24

     Yeah.

253. 31:24

     -right? That they can handle it.

254. 31:26

     Um, one thing we're looking at tomorrow that's launching is maybe you can sort of take over the VM and checkpoint the VM state and restart it.

255. 31:33

     Yep.

256. 31:33

     Um, uh, I think we have a dial-in, uh, call-in question from Paris. Um, if someone can play the video. Special guest.

257. 31:42

     Yeah. [laughs] Oh.

258. 31:44

     I wish I could be there to ask you in person. One of the questions that, that I have is, in this new world, the work-the workloads in the data center and the a- and the AI infrastructure is gonna be incredibly diverse.

259. 31:53

     On the one hand, uh, agents that are doing deep research, and they're thinking, they're reasoning, they're planning, and they're working with other agents, and they're, you know, working on a lot of memory.

260. 32:00

     They have large context on one hand. Um, some of it you also want it to think as fast as possible. So, you know, how, how do you, how do you create, uh, an AI infrastructure that is optimized for workloads that, uh, have to-- that a lot of prefill, a lot of decode, a lot of something in between

261. 32:12

     on the one hand? And on the other hand, uh, the type of workloads that I'm super excited about, these multimodal vision and speech AIs that are essentially your R2-D2, your companion that's on all the time, that's instantly available to you.

262. 32:23

     And so these two workloads, one of them, one of them super, uh, compute-intensive and take, might take a long time and, and, um, uh, you know, test time scaling and all that.

263. 32:32

     On the other hand, wants to be very low latency. So what does, what does a future AI infrastructure look like that's, that's as flexible as possible, um, as performant as possible, low latency, high throughput?

264. 32:41

     You know, all of that, uh, is just incredibly complex. So how, how do you think through that and, and, um, what kind of an AI infrastructure would you, would you think, uh, would be ideal going forward?

265. 32:48

     Okay. Well, with lot, lots of GPUs, of course. Um- [laughs]

266. 32:53

     So, so if I were to summarize, uh, Jensen wants you to tell him what to build. [laughs] [laughs]

267. 33:01

     What would be your dream? Uh, but also, like, there's just two needs. There's two kinds of infra. There's, there's long compute, and there's real-time now, now, now.

268. 33:08

     Yes. Yes, I mean, it's-- it, it is hard, right? Because, I mean,

269. 33:14

     this co-design problem, it is a mind-boggling one. And so, you know, I'm a software person by, by background, and that, you know, we think we're, we're off here just, like, writing the software for AGI, and then you realize you have to do, like, these massive infrastructure projects, right?

270. 33:26

     Like, that's not how we set out, but it actually kind of makes sense in the end, right? If we're going to build something that's going to be transformative to the world, like, yeah, probably it's gonna require some, some, you know, maybe the biggest physical machines that humanity's ever created.

271. 33:38

     Like, kind of type checks. Um, so I think that the-- that there's two answers. Like, the naive answer is, okay, yeah, you want two kinds of accelerators. You want one that's really compute optimized, one that's very latency optimized.

272. 33:50

     Um, throw, like, tons of, of HBM on one of those and, you know, ton-tons of, tons of compute on the other. You're all good. Um, now one thing that's really difficult is predicting the ratios, right?

273. 34:00

     Now you have a new problem you have to think about, and if you get the balance wrong, suddenly you're gonna have a whole part of your fleet that's just useless.

274. 34:05

     Yep.

275. 34:06

     And that sounds really scary. Um, now the thing is, because the way that these things work is there's no requirements in this field. There's no constraints in this field.

276. 34:13

     There's just sort of this linear program that people are optimizing. And so yeah, if you give our engineers some sort of misbalance of resources, like, we will find ways to utilize it, maybe at great pain, right?

277. 34:27

     But an example of this is, you know, you've seen the whole field move towards mixture of experts. And to some extent, what mixtur-mixture of experts is, is saying, "Well, we have all this DRAM sitting around that isn't being used for anything because the balance is wrong.

278. 34:38

     Fine, well, let's fill it up with parameters, and we'll actually not cost any compute, and we'll just get extra ML compute efficiency out of it." Like, boom, there you go.

279. 34:47

     And so I think that there is some of that, where if you get the balance wrong, it's actually not the end of the world. Um, homogeneity of accelerators is, like, a very nice default to start.

280. 34:55

     Um, but I think that, that, that ending up with purpose-built accelerators is also not super crazy. And the more that we move to these worl-this worlds where it's the just dollars of CapEx for this infrastructure starts to become so eye-watering, then starting to hyper-optimize for some of these workloads is pretty reasonable.

281. 35:12

     Um, but I think the jury's a little bit out, because if you think about it, that the research is just moving so fast, and to some extent, that dominates everything else.

282. 35:20

     Um, okay, I wasn't planning to ask this, but you just brought up the research stuff. Can you rank current scaling bottlenecks for GPT-6?

283. 35:27

     Ah. [laughs]

284. 35:28

     Compute, data, algorithms, power, money.

285. 35:31

     Yes. [laughs]

286. 35:34

     I mean, which one's, which one's, like, the, you know, number one and two? Which one are you, are you, like, most rate limited on?

287. 35:38

     I mean, look, I think we are in a world where basic research is back. I think that is really amazing, right? There was this period... Yeah. [laughs] Basic research. Um, there was a period where it felt like, all right, we got a transformer, let's just scale it.

288. 35:51

     You know, and, um, I find those problems very exciting. I have a lot of fun, just like you've got a very well-defined hard problem. You wanna just move the number up and to the right.

289. 36:00

     Um, but it also is a little intellectually dissatisfying in some ways. It's like that it feels like there's more to life than just, you know, attention is all you need paper-

290. 36:08

     Yeah.

291. 36:08

     -uh, you know, in, in, in vanilla form. Um, and so I think that what we've started to see is that we're operating at a scale now, um, where we've pushed the compute, we've pushed the data so far that you can start to get...

292. 36:23

     You start to have algorithms is like, again, just back as, as a important and really almost a long pole, um, in, in terms of f-future progress. And so, um, all of these things, they're all, they're all important poles of the tent and, you know, on any one day, uh, it might look a little lopsided one way or

293. 36:38

     another. Um, but yeah, fundamentally, I think it's like you wanna keep these all in balance. Um, and it's really exciting to see things like, like the RL paradigm. That's something that we invested in very deliberately-

294. 36:47

     Yeah.

295. 36:47

     -uh, for, for, for multiple years. It was like when we trained GPT-4, um, the very first thing, like, the thing that was really interesting was- When you, we talked to GPT-4 for the first time, we were like, "Is this an AGI?"

296. 36:59

     Like, it's clearly not an AGI, but it's really hard to say why, right? It's like there's something about it, it's so fluid and smooth, but, but somehow it falls off the rails.

297. 37:07

     And it's like, well, we gotta solve that reliability problem. And you're like, well, it has never actually experienced the world, right? It's like someone who's just read all the books or, you know, sort of read, you know, sort of observed the world, has observed the world, um, and, uh, never experienced it itself, right?

298. 37:22

     It's like, you know, sort of just, you know, watching it through, through a pane of glass or something. And, uh, and, and that to me is, I... You know, was, was something we were just like, "Okay, clearly we need a different paradigm," and we just pushed on it until we made it really work.

299. 37:36

     And I think that that remains true today, that there's other very clear missing capabilities, um, that we just need to keep pushing, and we will, we will get there.

300. 37:44

     Awesome. Um, broadening out just from, from just broad OpenAI things. Um, well, honestly, I'm just gonna let... So we asked Jensen for one question. He's a overachiever, so he sent in two.

301. 37:55

     So let's play the second video. [laughs]

302. 37:59

     The AI native engineers in the audience, they are probably thinking, um, uh, in the coming years, your OpenAI will have AGIs, and, uh, they will be building domain-specific agents on top of the AGIs from OpenAI.

303. 38:11

     And so some of the, some of the questions that I would have on my mind would be, uh, how you think their development workflow would change, uh, as, uh, OpenAI's AGIs become much more capable, and yet, uh, they would still have, um, a plumbing workflows, uh, pipelines that they would create, flywheels that they would create for their

304. 38:28

     domain-specific, uh, agents. Uh, these agents would, of course, uh, be able to reason, plan, use tools, have memory, short-term, long-term memory, and, and, um, uh, and, and they'll be amazing, amazing agents, but, uh, how does it change, uh, the development process, uh, in the coming years?

305. 38:40

     I guess-

306. 38:42

     Yeah, I think that this is a really fascinating question, right? I think you can find a wide spectrum of very strongly held opinion, uh, that is all mutually contradictory.

307. 38:51

     Um, I think my perspective is that, first of all, it's all on the table, right? Maybe we reach a world where it's just like the AIs are so capable, um, that, you know, we all, uh, you know, just let, let, let them write all the code.

308. 39:02

     Maybe there's a world where, that y- you have, like, one AI in the sky. Maybe it's that you actually have a bunch of domain-specific agents that require a bunch of, of specific work in order to make the, make it happen.

309. 39:13

     I think the evidence has really been shifting towards this, like, menagerie of different models. Um, and I think that's, that's actually really exciting, right? That there's different inference costs, just even from a systems perspective, um, that there's different trade-offs, like dis- distillation works so well.

310. 39:28

     Um, so there's actually a lot of power to be had by models that are actually able to use other models. And so I think that, that that is going to open up just a ton of opportunity because, you know, we're heading to a world where the economy is fundamentally powered by AI.

311. 39:41

     We're not there yet, but you can see it right on the horizon. And-

312. 39:44

     They're working on it all. [laughs]

313. 39:45

     Exactly. I mean, that's what people in this room are building. That is what you are doing. And the, the economy's a very big thing. There's a lot of diversity in it, and it's also not static, right?

314. 39:54

     That I think when people think about what AI can do for us, um, it's very easy to only look at, well, what are we doing now, and how does AI slot in and, you know, the percentage of human versus AI, but that's not the point, right?

315. 40:05

     The point is how do we get 10X more activity, 10X more economic output, 10X more benefit to everyone? Um, and I think that the direction we're heading is one where the models will get much more capable, there'll be much better fundamental technology, and there's just gonna be, like, way more things we wanna do with it, and the

316. 40:22

     barrier to entry will be lower than ever. And so things like healthcare, um, that you can't just... You know, the, the, it requires responsibility to go in and think about how to do it right.

317. 40:31

     Things like education, where there's multiple stakeholders, you know, the parent, the teacher, the student. Um, each of these requires domain expertise, requires careful thought, requires a lot of work.

318. 40:43

     Um, and so I think that there is going to be just, like, so much opportunity for people to build. Um, and so I'm just so excited to see everyone in this room because that's the right kind of energy.

319. 40:52

     Thank you for encouraging us and being an inspiration. Thank you so much.

320. 40:55

     Thank you.

321. 40:55

     Greg Brockman, everybody. [applause]

322. 40:57

     Thank you. [upbeat music]