See, Hear, Speak, Draw

Talk transcript

1. 0:00

   [upbeat music] So you can think of OpenAI as a, as a product and research company.

2. 0:17

   Uh, eh, we build awesome models and then we think about what are some of the best ways to apply them to solve the biggest problems that humanity faces. And so there's this, uh, deployment pipeline.

3. 0:29

   Logan and I sit at the end of this deployment pipeline. We work with, uh, people in the real world that are using OpenAI's models. We spend our time thinking about what are some of the best ways to use our models, what are some of the hardest problems that haven't been solved yet, and how can we apply, uh,

4. 0:45

   OpenAI technology to solve these? Um, I'm on the apply team, and I'm a engineer.

5. 0:51

   Yeah. And my name's Logan Kilpatrick, and I do developer relation stuff, so helping people build fun and exciting, uh, products and services using our, using our API.

6. 1:05

   So yeah, folks saw from the title of the talk, uh, we'll, we'll talk about multimodal stuff, but I think it's important to start off with, with where we are today.

7. 1:14

   And I think, you know, as, as we all know, people who have been building in the AI space for the last six, 12, 18 months, um, 2023 has really been the year of chatbots.

8. 1:22

   And I think it's, uh, it's been incredible to see how much people have actually been able to do, how much value you can create in the world with, like, just a simple chatbot.

9. 1:31

   Um, and it's, i- it still blows my mind to think about how rudimentary these systems are and how much more value that's gonna be created, um, in the next year, in the next decade.

10. 1:41

    Um, and that's why I'm excited for 2024, which I think is, is really gonna be the... I don't know if I can trademark this, but the year of multimodal models.

11. 1:49

    Um, it's a tongue twister, but also- [laughs] ... hopefully the domain is available, yearofmultimodals.com. Um- [laughs] No, don't, don't buy it if it's available. [laughs] [laughs] Uh, yeah, so I'm, I'm excited. We-- OpenAI has a ton of multimodal capabilities that are, that are in the works.

12. 2:05

    Um, some folks might have already tried some of these in ChatGPT in the iOS app or the, the web app today, things like, uh, vision, taking in images, describing them.

13. 2:13

    Um, we'll, we'll show that later on. Um, also the ability to generate images. We've had this historically with, with DALL·E 2, but, uh, DALL·E 3 really... If, if folks have tried it, it, it takes things to the next level, so excited to, to show some of that today as well.

14. 2:29

    Cool. So if you, if you think of, uh, the way that multimodal capabilities are working right now, eh, it's a little bit of a, of a setup of islands where we have DALL·E that takes texts and generates images.

15. 2:43

    We have Whisper that takes an audio and gen- generates text transcripts. Uh, we have a GPTV with vision capabilities, GPT-4 with Vision capabilities that takes images and text and can reason over both at the same time.

16. 2:56

    Um, but right now these, these are all very disparate things. Uh, eh, however, you can think of text as a connective tissue between all of these models. Eh, and there's a lot of interesting things that, that, uh, we can build right now, uh, eh, using that paradigm.

17. 3:16

    But what we're actually really excited for is, uh, a future in which there's unity between all these modalities and, eh... And this is where we're going. This is not where we're today.

18. 3:28

    Uh, but you can, you can think of models in the same way that, like, GPT can consume, uh, images and text simultaneously. Uh, maybe in the future we'll consume even more modalities and we'll output even more modalities and we'll be able to reason about them in the, at the same time.

19. 3:41

    However, we're not there yet. And so, uh, today Logan and I are gonna show you just, like, some, some, uh, architecture patterns and some ways in which you can, uh, mimic this kind of situation with what we have available today, eh, and, and, and some of the patterns that you can start to think about as we move

20. 3:59

    towards this future in which models can, eh, reason way beyond text.

21. 4:05

    As Simón and I were, were making these demos today, um, waiting till the, the last minute as, as always, it, it was really interesting to see that, like, really much of the work of making multimodal systems today is, like, how do you hook everything up together and connect the different modalities?

22. 4:20

    And again, as Simón said, using text as sort of the, the bridge between different modalities. Um, but it, it's gonna be super interesting to see, like, how much developer efficiency gains there are when you no longer have to do that and you really just have, like, a single model that can do text in, text out, video at

23. 4:34

    some point, you know, speech in, speech out at some point. Um, so it'll be super cool to, to see when that's possible and, uh, make, make, making demos even, uh, even easier and simpler.

24. 4:45

    Cool. All right. Well, uh, we'll show you guys two demos today. Uh, a- and we'll talk about, like, some, some high-level ideas and some high-level concepts. Uh, and hopefully at the end of it you'll, you'll be inspired to think about, like, well, what are some of the things that, uh, maybe you're not able to build today, but

25. 5:00

    you'll be able to build, uh, six months, a year from now, eh, and how you should start thinking about your products, uh, um, uh, as they are able to incorporate more, uh, modalities.

26. 5:11

    Cool. So on to demo number one. Uh, this is a, it's a, it's a very, very simple DALL·E Vision loop. Um-

27. 5:22

    Yeah

28. 5:22

    ... do you wanna walk us through?

29. 5:22

    So yeah, [laughs] sorry. Um, excited to, to look at this demo. So Simón will, will pull up the demo, and I'll, I'll sort of just walk through. But the basic idea is let's take a real image.

30. 5:32

    Um, let's use GPTV, um, or GPT-4 with, with image inputs to essentially create a nice human, uh, readable, understandable description of that image. Um, and then we'll put that into DALL·E 3 and actually go and generate a synthetic version of that image.

31. 5:48

    Um, so this whole pipeline takes a little bit to run because, uh, it's not a production, um, system at the moment. Uh, but the nice part is, uh, we've got a couple of examples ready and we can...

32. 6:01

    You wanna kick one off live as well? [laughs] We can-

33. 6:03

    Cool. Okay

34. 6:04

    ... let it run in the background. [laughs]

35. 6:05

    So very, very-- this is a, a fun, simple idea. But, uh, the, this is a, a photo that I took in the lobby downstairs, uh, just when you walk into the, the hotel.

36. 6:16

    Uh, there are these, uh, kind of like, uh, Halloween-themed painted ladies. Um, uh, and so what we did here is that we asked, uh, GPT-4 with Vision to describe this image in detail Uh, and then we asked it to, uh, generate, uh, a description for DALL·E to, uh, generate a new image based on this.

37. 6:36

    Um, uh, you can see it, it does an okay job. Here's a description of the image. Here's a prompt it uses. Uh, it, it picks up on a lot of details like the, the RIP in the tombstone and the all dogs, uh, welcome, uh, thing here, and then it generates a whole new image.

38. 6:52

    But there's a lot of details that are off, you know? Like, the, the, um, the marble is black and the, uh, the spiders are white. Eh, and so what we do next is that we pass the-

39. 7:05

    It's close.

40. 7:05

    Yeah, it's close enough.

41. 7:06

    It's close. [laughs]

42. 7:06

    Uh, [laughs] but we give the two images to the GPT with Vision again, um, and we ask it to compare them, uh, and see what are some of the differences.

43. 7:15

    And, uh, it, it picks up on a lot of the, the different details. And then we ask it to create a new, a new image based on these, uh, differences, and it goes ahead.

44. 7:25

    New image. You see. [laughs] [laughs] Uh, all the black marble is gone. The spider is now larger [laughs] and black. [laughs] Uh, but you know, it, it, it matches them closely. And I think this, this is just to illustrate.

45. 7:38

    I think there's a long way to go, but this is to illustrate the idea that there are plenty of tasks that we do right now in AI where we, we

46. 7:47

    need the human in the loop to be able to evaluate a visual output that a model produces, compare it with something else, then, like, iterate on the instructions, pass that again to another model.

47. 7:56

    And so th- that, that's a pipeline where we, like, thought that humans were very essential and that we're probably gonna continue to be essential for some time. And now that's something that, uh, the models can do by themselves.

48. 8:05

    Um, a- and there's a couple of, uh, of, uh, interesting, uh,

49. 8:09

    uh, patterns here, I think. I think one of them is describing images. That's powerful because now you have an image. Y- now you have text, and you can reason about that text.

50. 8:16

    You can do a lot of things with that text. But another really powerful element is, like, uh, comparing images, um, uh, and, and, and spotting differences, uh, like having, like, a final destination that you want to get to and, like, a current destination.

51. 8:29

    And, uh, a- and that pattern of comparing things, you can apply to a lot of things. So imagine, uh, talking now. We're, we're-- Logan and I were just chatting about, like, some other, like, ways that you can apply this and, and, and, uh, Logan's idea was imagine you are, uh, curating, uh, your room and you're, you just

52. 8:43

    moved to a new place. Uh, and you're on Instagram, and you find some images of, like, a vibe that you like and, and like, maybe some object. And, and then you can grab that image.

53. 8:52

    You can give that to GPT-4 with Vision, and you can tell it, like, "Okay, now, like, like, crawl through Amazon and find, like, all the lamps that match this vibe that I want for my room."

54. 9:01

    Um-

55. 9:02

    I want this so badly.

56. 9:03

    Yeah.

57. 9:03

    This would be so cool.

58. 9:03

    And, and, and so [laughs]

59. 9:04

    I can't do interior design, so it's like I, I would love to be able to just be like, "Get me all the stuff that matches this specific vibe." It's, it's a, a hard problem right now.

60. 9:12

    Yeah. Um, and a couple-

61. 9:14

    Simón, can I make one other quick comment, which is just-

62. 9:16

    For sure

63. 9:16

    ... I, I think also, you know, folks were, were laughing, you know, in, in good jest when this, when this third image came up, came up. I think it's important to know that there's, there's, like, no prompt engineering or anything like that that's happening.

64. 9:27

    This is, like, the, the rawest output that you can get. This is a, a one-hour demo version. So people can, uh, will hopefully go wild with this once it's available through the API and, like, ideally get much better results than, than we're seeing today.

65. 9:40

    Um, yeah.

66. 9:41

    Yeah.

67. 9:41

    Probably using a bunch of techniques that other people talked about at the conference so far.

68. 9:44

    Right.

69. 9:44

    So this is the, the very basic version of, of this demo.

70. 9:48

    Yeah. And, and we wanted to keep it simple and minimal just to illustrate the, the, the power of the models. This is as raw as you can get when it comes to the models.

71. 9:56

    The, like, there is almost all the completion output is going straight into the model, and, and I think there's, like, 50 lines of code. So, like, the majority of the power lifting is being done by the models here.

72. 10:06

    Um, um, another quick example that I'll show you guys, and then I'll try to do one live, uh, which will probably, uh, be tragic. But, uh, [laughs]

73. 10:17

    um, so this is, this is the backstage right here. [laughs] [laughs]

74. 10:22

    Uh, uh, I just took this photo right before walking on stage. Uh, uh, you can see that, uh, GPT with Vision does a really good job actually of describing that.

75. 10:30

    Uh, the, there's the monitors, and there's boxes, and there's cables, and there's whatnot. Uh, uh, and then this is the image that DALL·E generates, DALL·E 3. Uh, so you can see blue carpet, cables, boxes, all the elements.

76. 10:44

    And then it goes on to spot the differences, and it notices, for example, that in this image, there are all these vertical lights that are not present in the first image.

77. 10:52

    It says that here. Uh, lighting. Like, all this, like, vertical lights on the walls and ceilings which adds... Uh, but then it rewrites the prompt, and it gets rid of all the vertical lights. [laughs] [laughs]

78. 11:05

    And it g- and it, and it adds the, uh, the curtain in the back, which wasn't present here but is present in the, the black curtain here. Um, so little, l- just little interesting things.

79. 11:14

    It's still a long way to go, but, like, uh, this, this new, this whole new... This opens a whole new box of interaction patterns, the, the, the fact that now you can reason visually.

80. 11:22

    Um, cool. And, and let's give a shot to, uh, a live example. So this, uh, this was a, a trail run that I did over the weekend, uh, up in [REDACTED:location].

81. 11:34

    Um, and so I'm just gonna do it from scratch. Um,

82. 11:41

    hope that it works. [laughs]

83. 11:45

    If you wanna go to another-

84. 11:46

    There you go. [laughs] Uh, cool. So the image depicts a, a serene and picturesque woodland setting. Uh, the focus of the image is a wooden boardwalk or a footbridge that winds through the dense forest.

85. 11:59

    Uh, very detailed description. Light filters through the trees. [laughs] Uh, and I'm just passing that raw just straight to DALL·E.

86. 12:10

    Yeah. And if, if folks have seen what happens in the, the DALL·E, uh, mode in the ChatGPT iOS app, for example, it's actually doing a little bit. I, I don't, uh, know off the top of my head, like, what the prompt is for that, but it's, it's doing some amount of prompt engineering.

87. 12:25

    Like, if folks have actually tried to use, like, our Labs product before to make DALL·E images, you have to do that prompt engineering yourself. Um, and I think that's been one of the limitations.

88. 12:34

    Like, if people used Midjourney or, or other, um- Other image models in the past, like it's just kind of hard to make good prompts that work well for these systems, so it's nice that the, uh, the model can, can take a stab at doing it for you.

89. 12:51

    It's telling us a lot of how the, the second image is a lot more beautiful and more detailed [laughs] [laughs] Which checks out. [applause]

90. 13:10

    It's, it's also interesting to see, uh, just for folks to, to think about. It's interesting to see that like it's still of these, um, image models, like the main limitation as we're seeing-

91. 13:20

    I know

92. 13:20

    ... this demo in real time is actually- [laughs]

93. 13:24

    No. [laughs] [laughs] Of course. [laughs] [applause] Here you go.

94. 13:33

    We're going back to the slides. Next... Go back a slide to where we have, you know, the time.

95. 13:37

    All right. I'm gonna leave it running, and then at, at the time if we, if we have time it'll probably work the second round because-

96. 13:43

    It worked the three times before this [laughs]

97. 13:46

    Um, cool. Okay. For the second demo, um, uh, we're gonna take it a little bit further, and we're gonna do something, uh, w- with video. Uh, and the idea here is that there's a lot of video summarization demos out there that we've seen.

98. 14:06

    Uh, the majority of them just take a transcript and then, uh, ask GPT-4 to summarize this transcript. However, videos have a lot of, uh, information in them that is conveyed visually.

99. 14:16

    And so, uh, what we're doing here is that we're taking frames from the video, um, and then we're asking GPT-4 with Vision to describe all the frames, and then we are asking Whisper to transcribe the video.

100. 14:29

     And now we have this long textual representation of the video that not only includes all the audio information, but also includes visual information from the video. And then we're doing some exciting like mixes on that, uh, that Logan will tell you about.

101. 14:42

     Yeah. I'm ready for the next slide. Um, yeah, so for, for this demo we're literally just taking the GPT-4 introduction, uh, video folks have seen on YouTube. It's a good video if you haven't seen it before.

102. 14:55

     Um, so taking the video raw from YouTube. Go to the next slide. Uh, taking the video raw from YouTube, again, like Simón said, cutting up those, uh, the different frames from the video, putting those into, to GPT-4 with image input, getting the summaries which you can see, and I know it's really hard.

103. 15:12

     Um, but literally just like actually saying what's... Th- these are simple images, so it's easy to capture the, the depth of what's shown here. Um, taking those images and then going to the next piece which is essentially a big, another, another wonderful J- DALL·E image, but a big description of, uh, of the transcript and then all of

104. 15:34

     the image... Essentially like image embeddings is the, is the easiest way of thinking about it. So if you wanna actually see the results of this, QR code bottom right-hand corner is real.

105. 15:44

     Um, you can scan it and see the resulting article. It's, it's pretty- [laughs] [laughs]

106. 15:52

     It's pretty good. Um, it does a good job, and I think for, for me, you know, why this is exciting is 'cause you can sort of capture the, again, capture the depth of, uh, of what happens in a video.

107. 16:04

     So a DALL·E image to start, and then a bunch of actual frames that like match up with the contextual representation of what's being talked about in the blog post.

108. 16:13

     Um, and again, there's no hand... I, I couldn't open source the code 'cause it has a bunch of unreleased APIs, but no, no sort of magic behind the scenes stuff that's happening.

109. 16:24

     This is like a raw, crappy prompt, um, to generate this, uh, this blog post. Which I think is, again, I think it's really cool and, um, takes videos and, and makes them more accessible in, in the, in the text form.

110. 16:38

     So I like it.

111. 16:39

     Cool. Let's see if this finished. No.

112. 16:46

     Oh. [laughs]

113. 16:47

     Oh, well. Um... [laughs] Cool.

114. 16:55

     Can you bring it back a sec?

115. 16:55

     Oh, yeah. Sure. Cool. [laughs] [laughs] Okay. Uh, so some, some, uh, concluding takeaways.

116. 17:11

     Um, uh, start thinking multimodal. Uh, that's, that's something net new that's, that's happening these days and, and if you have any crazy ideas that you think, "Wow, it would be really cool if, if technology could do this," uh, we'll probably be able to get there and, and the products that you'll be able to build six months from

117. 17:30

     now, a year from now are gonna be incredible. So start having this in mind as, as, as people who are building AI products and people who are building companies.

118. 17:39

     Um, think of text as a, as a connecting tissue right now. Uh, and, and I think this is a very powerful concept, and that's gonna continue to be the case for the near future.

119. 17:49

     Uh, uh, and there are many powerful patterns that are yet to be explored when it comes to multimodal stuff, especially when it comes to, to, uh, doing things with images.

120. 17:57

     Uh, so really excited to, uh, soon get this in the hands of all of you guys and, and to see what you all build with this. I think it's, uh, it's really exciting, uh, to see, uh, AI start to venture into the visual world.

121. 18:12

     Yeah, agents with image input is gonna be sick. I can't wait.

122. 18:15

     Yeah.

123. 18:15

     I feel like so much of the internet is, requires that. Yeah, and we're excited. I think there's, there's a lot of stuff that's gonna happen in the, in the near future and, um, I think it's cool to be able to hopefully get a glimpse of, of what some of those use cases look like.

124. 18:27

     So anything else you wanna say, Simón?

125. 18:30

     No, that's good.

126. 18:31

     All right.

127. 18:31

     Yeah.

128. 18:31

     This was wonderful. Thank you all. [applause]

129. 18:33

     Thank you all. [upbeat music]