AI Engineer Summit 2023
See, Hear, Speak, Draw
Read the talk
See, Hear, Speak, Draw: Connecting Multimodal Models Through Text
An image reconstruction loop and a video-to-article pipeline show how text can connect visual understanding, speech transcription, and image generation.
From a talk by Logan Kilpatrick and Simón Fishman
Before you start: Basic familiarity with language-model prompts and passing one model’s output into another is helpful.
What can you build beyond a chatbot?
How do you turn a research model into an application that solves a real problem? At OpenAI, Simón Fishman and Logan Kilpatrick work at that boundary: Fishman as an Applied engineer working with model users and unsolved problems, Kilpatrick in developer relations helping developers build products with the API. Their starting point is a company that both develops models and finds ways to put them to work.
Chatbots have already shown how much value a simple interface can deliver. Speaking in 2023, Kilpatrick calls it the year of chatbots and predicts that 2024 will be the year of multimodal models. The immediate examples are concrete: users are beginning to try image understanding in ChatGPT’s image-capability rollout, and image generation is advancing from DALL·E 2 to DALL·E 3. The question becomes what developers can build when their applications can work with more than typed conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Text connects the model islands
The architecture starts with separate models, each handling a different conversion.
| Model | Input | Output used by the application |
|---|---|---|
| DALL·E | Text | Generated image |
| Whisper | Audio | Transcript |
| GPT-4 with Vision, also called GPT-V | Images and text | Text describing or reasoning about the images |
Text is the shared interface. A transcript can become material for another model to reason over; an image description can become a generation prompt. The models need not share a native representation for the application to connect them.
Fishman distinguishes this composition from the future he wants: a model that can consume and produce more modalities while reasoning across them together. In the architecture demonstrated here, developers still do the wiring. Kilpatrick says that connecting modalities accounted for much of the work of building the demos; a unified model accepting text, video, or speech could remove some of that integration work. The two examples that follow explore what developers can assemble with separate models while thinking ahead to products they might build over the next six months or year. That horizon is an invitation to plan, not a release commitment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Describe, generate, compare, revise
The first demonstration is a DALL·E Vision loop. Start with a real photograph, ask GPT-4 with Vision for a readable description, then send that description to DALL·E 3 to generate a synthetic version. The pipeline takes enough time that the speakers bring prepared examples and also attempt a live run.
Fishman’s first photograph comes from the hotel lobby: a display of Halloween-themed painted ladies. Vision describes the scene and produces a prompt for DALL·E, picking up details including RIP on a tombstone and a sign welcoming all dogs. The generated image captures parts of the scene but introduces conspicuous differences: black marble and white spiders.
The application then closes the loop:
- Give GPT-4 with Vision both the original photograph and the generated image.
- Ask it to identify differences between them.
- Use those differences to produce revised generation instructions.
- Generate another image with DALL·E.
The next image removes the black marble and makes the spider black, although it also makes the spider larger. The result is still imperfect, but it demonstrates an operation beyond describing an isolated picture: compare a current result with a target, then turn the discrepancy into instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make visual evaluation part of the application
This loop resembles work a human would otherwise perform: inspect a visual output, compare it with a reference, revise the instructions, and run the model again. Fishman acknowledges that there is a long way to go, but the demonstration moves some of that evaluation into the model pipeline. Two operations matter:
- Description: Convert an image into text that subsequent reasoning steps can consume.
- Comparison: Examine the difference between a current state and a desired state, then use that difference to guide the next action.
The second operation gives the application a target to work toward rather than merely another caption to produce.
An imagined interior-design application extends the comparison pattern. Someone moving into a new room finds an Instagram image with the atmosphere they want. They could provide that reference and ask an application to search Amazon for lamps that match it. Here the desired result is not an exact reconstruction: it is a set of objects compatible with the reference’s visual style. Fishman and Kilpatrick discuss this as a product idea, not as a shopping crawler they have built.
The roughness of the lobby result also reflects a deliberately minimal implementation. Kilpatrick describes a one-hour demo with no prompt engineering; Fishman estimates roughly 50 lines of code for the image loop. Almost all completion output goes straight into the next model. They expect additional techniques to improve the results, but do not measure that improvement here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remove invented lights and restore a curtain
A second prepared example uses a photograph Fishman took backstage immediately before the talk. Vision describes monitors, boxes, and cables; DALL·E’s reconstruction includes blue carpet, cables, and boxes. But the generated room also contains vertical lights that were not in the photograph.
The comparison identifies those extra lights on the walls and ceiling. After the prompt is rewritten, the next generation removes them and adds the black curtain visible at the rear of the original scene. These are specific, observable corrections, even though the whole room is not faithfully reconstructed. The useful capability is that a visual discrepancy can become an instruction without a person first spelling out what is wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A live woodland run exposes the limits
For the live attempt, Fishman selects a trail-running photograph from Purisima Woods. Vision describes a wooden boardwalk or footbridge winding through dense forest, with light filtering through the trees. He passes that description directly to DALL·E.
While it runs, Kilpatrick explains another useful role for language models: writing image prompts. ChatGPT’s DALL·E mode does some prompt engineering for the user, although he does not know its exact prompt. In the earlier Labs workflow, users had to do that work themselves. He points to Midjourney and other image models as further examples of systems for which writing effective prompts can be difficult. A language model can mediate between what a person wants and the instructions an image generator needs.
The woodland comparison describes the generated image as more beautiful and detailed. That is a different judgment from whether it matches the photograph: an attractive reconstruction can still be an inaccurate one. The live attempt then runs into trouble, and the speakers return to the slides while leaving it running. Kilpatrick reports three successful runs before the onstage attempt, but the demonstration has not yet reached a completed result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A video contains more than its transcript
The second demonstration addresses a limitation of transcript-based video summaries. Sending only the spoken words to GPT-4 leaves out information conveyed visually. Instead, the application extracts frames and asks GPT-4 with Vision to describe them, while Whisper transcribes the audio. The two streams become a long textual representation of what the video says and shows.
The worked input is OpenAI’s GPT-4 introduction video from YouTube. Its relatively simple frames are easy to describe in text. Those descriptions join the transcript as source material for an article. Kilpatrick informally calls the visual material “image embeddings,” but the operation he describes is textual frame description, not an explicit vector-embedding step.
The assembly step can remain simple. In Python, keep the transcript and frame descriptions explicitly separated so the downstream writing prompt can refer to both:
python
def article_prompt(transcript: str, frame_descriptions: list[str]) -> str:
frames = "\n\n".join(
f"Frame {index}:\n{description}"
for index, description in enumerate(frame_descriptions, start=1)
)
return (
"Write an article using both the spoken content and visual details.\n"
"Indicate which supplied frames belong beside relevant passages.\n\n"
f"TRANSCRIPT\n{transcript}\n\n"
f"FRAME DESCRIPTIONS\n{frames}"
)
This is the text handoff: descriptions let the writing model use visual information, while references to the supplied frames let the application retain the actual pictures.
The resulting article, Don’t forget about GPT-4, begins with a DALL·E image and includes actual video frames matched to the surrounding discussion. The output therefore offers two kinds of access to the source: prose that incorporates visual details, and images that preserve the source’s appearance.
Kilpatrick says the article was generated with a raw prompt, without hidden manual work, and that he could not open-source the code because it used unreleased APIs. This is a 2023 demonstration, not a current API recipe: ChatGPT’s image rollout was already underway, while public GPT-4 vision and DALL·E 3 API access followed in the November 2023 DevDay announcement. The product goal is straightforward—make video content accessible in text without throwing away everything the viewer could see.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Products that can work with what they see
Before closing, Fishman checks the woodland run again. It still has not finished. The prepared image examples and the illustrated article remain the completed demonstrations; the live run does not acquire a successful ending.
Fishman’s closing advice is to start thinking about products that incorporate multiple modalities, including ideas that still feel out of reach. His six-month-to-one-year outlook is optimistic, while his immediate architectural guidance remains practical: text will continue to connect models, and image-based interaction patterns offer substantial room for exploration.
Kilpatrick ends with image-input agents. Much of the internet requires understanding what is visible, so an agent that can interpret images can engage with information unavailable through text alone. The demos provide an early vocabulary for that work: describe what is present, compare it with what is wanted, and use the resulting text to decide what comes next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Logan Kilpatrick’s generated Summit demo article, combining a DALL·E 3 cover with frames from its source video.
OpenAI’s speech-recognition code, with installation and transcription examples.
An overview of DALL·E 3 image generation and ChatGPT-assisted prompt refinement.
Further reading
OpenAI’s introduction to its evaluation and mitigation work for GPT-4 image inputs, with a link to the system card.
The September 2023 announcement of ChatGPT image and voice capabilities and their staged rollout.
Updates since the talk
The November 2023 release announcement for GPT-4 vision inputs, DALL·E 3 image generation and text-to-speech APIs.
Read the complete timestamped transcript
- 0:00
[upbeat music] So you can think of OpenAI as a, as a product and research company.
- 0:17
Uh, eh, we build awesome models and then we think about what are some of the best ways to apply them to solve the biggest problems that humanity faces. And so there's this, uh, deployment pipeline.
- 0:29
Logan and I sit at the end of this deployment pipeline. We work with, uh, people in the real world that are using OpenAI's models. We spend our time thinking about what are some of the best ways to use our models, what are some of the hardest problems that haven't been solved yet, and how can we apply, uh,
- 0:45
OpenAI technology to solve these? Um, I'm on the apply team, and I'm a engineer.
- 0:51
Yeah. And my name's Logan Kilpatrick, and I do developer relation stuff, so helping people build fun and exciting, uh, products and services using our, using our API.
- 1:05
So yeah, folks saw from the title of the talk, uh, we'll, we'll talk about multimodal stuff, but I think it's important to start off with, with where we are today.
- 1:14
And I think, you know, as, as we all know, people who have been building in the AI space for the last six, 12, 18 months, um, 2023 has really been the year of chatbots.
- 1:22
And I think it's, uh, it's been incredible to see how much people have actually been able to do, how much value you can create in the world with, like, just a simple chatbot.
- 1:31
Um, and it's, i- it still blows my mind to think about how rudimentary these systems are and how much more value that's gonna be created, um, in the next year, in the next decade.
- 1:41
Um, and that's why I'm excited for 2024, which I think is, is really gonna be the... I don't know if I can trademark this, but the year of multimodal models.
- 1:49
Um, it's a tongue twister, but also- [laughs] ... hopefully the domain is available, yearofmultimodals.com. Um- [laughs] No, don't, don't buy it if it's available. [laughs] [laughs] Uh, yeah, so I'm, I'm excited. We-- OpenAI has a ton of multimodal capabilities that are, that are in the works.
- 2:05
Um, some folks might have already tried some of these in ChatGPT in the iOS app or the, the web app today, things like, uh, vision, taking in images, describing them.
- 2:13
Um, we'll, we'll show that later on. Um, also the ability to generate images. We've had this historically with, with DALL·E 2, but, uh, DALL·E 3 really... If, if folks have tried it, it, it takes things to the next level, so excited to, to show some of that today as well.
- 2:29
Cool. So if you, if you think of, uh, the way that multimodal capabilities are working right now, eh, it's a little bit of a, of a setup of islands where we have DALL·E that takes texts and generates images.
- 2:43
We have Whisper that takes an audio and gen- generates text transcripts. Uh, we have a GPTV with vision capabilities, GPT-4 with Vision capabilities that takes images and text and can reason over both at the same time.
- 2:56
Um, but right now these, these are all very disparate things. Uh, eh, however, you can think of text as a connective tissue between all of these models. Eh, and there's a lot of interesting things that, that, uh, we can build right now, uh, eh, using that paradigm.
- 3:16
But what we're actually really excited for is, uh, a future in which there's unity between all these modalities and, eh... And this is where we're going. This is not where we're today.
- 3:28
Uh, but you can, you can think of models in the same way that, like, GPT can consume, uh, images and text simultaneously. Uh, maybe in the future we'll consume even more modalities and we'll output even more modalities and we'll be able to reason about them in the, at the same time.
- 3:41
However, we're not there yet. And so, uh, today Logan and I are gonna show you just, like, some, some, uh, architecture patterns and some ways in which you can, uh, mimic this kind of situation with what we have available today, eh, and, and, and some of the patterns that you can start to think about as we move
- 3:59
towards this future in which models can, eh, reason way beyond text.
- 4:05
As Simón and I were, were making these demos today, um, waiting till the, the last minute as, as always, it, it was really interesting to see that, like, really much of the work of making multimodal systems today is, like, how do you hook everything up together and connect the different modalities?
- 4:20
And again, as Simón said, using text as sort of the, the bridge between different modalities. Um, but it, it's gonna be super interesting to see, like, how much developer efficiency gains there are when you no longer have to do that and you really just have, like, a single model that can do text in, text out, video at
- 4:34
some point, you know, speech in, speech out at some point. Um, so it'll be super cool to, to see when that's possible and, uh, make, make, making demos even, uh, even easier and simpler.
- 4:45
Cool. All right. Well, uh, we'll show you guys two demos today. Uh, a- and we'll talk about, like, some, some high-level ideas and some high-level concepts. Uh, and hopefully at the end of it you'll, you'll be inspired to think about, like, well, what are some of the things that, uh, maybe you're not able to build today, but
- 5:00
you'll be able to build, uh, six months, a year from now, eh, and how you should start thinking about your products, uh, um, uh, as they are able to incorporate more, uh, modalities.
- 5:11
Cool. So on to demo number one. Uh, this is a, it's a, it's a very, very simple DALL·E Vision loop. Um-
- 5:22
Yeah
- 5:22
... do you wanna walk us through?
- 5:22
So yeah, [laughs] sorry. Um, excited to, to look at this demo. So Simón will, will pull up the demo, and I'll, I'll sort of just walk through. But the basic idea is let's take a real image.
- 5:32
Um, let's use GPTV, um, or GPT-4 with, with image inputs to essentially create a nice human, uh, readable, understandable description of that image. Um, and then we'll put that into DALL·E 3 and actually go and generate a synthetic version of that image.
- 5:48
Um, so this whole pipeline takes a little bit to run because, uh, it's not a production, um, system at the moment. Uh, but the nice part is, uh, we've got a couple of examples ready and we can...
- 6:01
You wanna kick one off live as well? [laughs] We can-
- 6:03
Cool. Okay
- 6:04
... let it run in the background. [laughs]
- 6:05
So very, very-- this is a, a fun, simple idea. But, uh, the, this is a, a photo that I took in the lobby downstairs, uh, just when you walk into the, the hotel.
- 6:16
Uh, there are these, uh, kind of like, uh, Halloween-themed painted ladies. Um, uh, and so what we did here is that we asked, uh, GPT-4 with Vision to describe this image in detail Uh, and then we asked it to, uh, generate, uh, a description for DALL·E to, uh, generate a new image based on this.
- 6:36
Um, uh, you can see it, it does an okay job. Here's a description of the image. Here's a prompt it uses. Uh, it, it picks up on a lot of details like the, the RIP in the tombstone and the all dogs, uh, welcome, uh, thing here, and then it generates a whole new image.
- 6:52
But there's a lot of details that are off, you know? Like, the, the, um, the marble is black and the, uh, the spiders are white. Eh, and so what we do next is that we pass the-
- 7:05
It's close.
- 7:05
Yeah, it's close enough.
- 7:06
It's close. [laughs]
- 7:06
Uh, [laughs] but we give the two images to the GPT with Vision again, um, and we ask it to compare them, uh, and see what are some of the differences.
- 7:15
And, uh, it, it picks up on a lot of the, the different details. And then we ask it to create a new, a new image based on these, uh, differences, and it goes ahead.
- 7:25
New image. You see. [laughs] [laughs] Uh, all the black marble is gone. The spider is now larger [laughs] and black. [laughs] Uh, but you know, it, it, it matches them closely. And I think this, this is just to illustrate.
- 7:38
I think there's a long way to go, but this is to illustrate the idea that there are plenty of tasks that we do right now in AI where we, we
- 7:47
need the human in the loop to be able to evaluate a visual output that a model produces, compare it with something else, then, like, iterate on the instructions, pass that again to another model.
- 7:56
And so th- that, that's a pipeline where we, like, thought that humans were very essential and that we're probably gonna continue to be essential for some time. And now that's something that, uh, the models can do by themselves.
- 8:05
Um, a- and there's a couple of, uh, of, uh, interesting, uh,
- 8:09
uh, patterns here, I think. I think one of them is describing images. That's powerful because now you have an image. Y- now you have text, and you can reason about that text.
- 8:16
You can do a lot of things with that text. But another really powerful element is, like, uh, comparing images, um, uh, and, and, and spotting differences, uh, like having, like, a final destination that you want to get to and, like, a current destination.
- 8:29
And, uh, a- and that pattern of comparing things, you can apply to a lot of things. So imagine, uh, talking now. We're, we're-- Logan and I were just chatting about, like, some other, like, ways that you can apply this and, and, and, uh, Logan's idea was imagine you are, uh, curating, uh, your room and you're, you just
- 8:43
moved to a new place. Uh, and you're on Instagram, and you find some images of, like, a vibe that you like and, and like, maybe some object. And, and then you can grab that image.
- 8:52
You can give that to GPT-4 with Vision, and you can tell it, like, "Okay, now, like, like, crawl through Amazon and find, like, all the lamps that match this vibe that I want for my room."
- 9:01
Um-
- 9:02
I want this so badly.
- 9:03
Yeah.
- 9:03
This would be so cool.
- 9:03
And, and, and so [laughs]
- 9:04
I can't do interior design, so it's like I, I would love to be able to just be like, "Get me all the stuff that matches this specific vibe." It's, it's a, a hard problem right now.
- 9:12
Yeah. Um, and a couple-
- 9:14
Simón, can I make one other quick comment, which is just-
- 9:16
For sure
- 9:16
... I, I think also, you know, folks were, were laughing, you know, in, in good jest when this, when this third image came up, came up. I think it's important to know that there's, there's, like, no prompt engineering or anything like that that's happening.
- 9:27
This is, like, the, the rawest output that you can get. This is a, a one-hour demo version. So people can, uh, will hopefully go wild with this once it's available through the API and, like, ideally get much better results than, than we're seeing today.
- 9:40
Um, yeah.
- 9:41
Yeah.
- 9:41
Probably using a bunch of techniques that other people talked about at the conference so far.
- 9:44
Right.
- 9:44
So this is the, the very basic version of, of this demo.
- 9:48
Yeah. And, and we wanted to keep it simple and minimal just to illustrate the, the, the power of the models. This is as raw as you can get when it comes to the models.
- 9:56
The, like, there is almost all the completion output is going straight into the model, and, and I think there's, like, 50 lines of code. So, like, the majority of the power lifting is being done by the models here.
- 10:06
Um, um, another quick example that I'll show you guys, and then I'll try to do one live, uh, which will probably, uh, be tragic. But, uh, [laughs]
- 10:17
um, so this is, this is the backstage right here. [laughs] [laughs]
- 10:22
Uh, uh, I just took this photo right before walking on stage. Uh, uh, you can see that, uh, GPT with Vision does a really good job actually of describing that.
- 10:30
Uh, the, there's the monitors, and there's boxes, and there's cables, and there's whatnot. Uh, uh, and then this is the image that DALL·E generates, DALL·E 3. Uh, so you can see blue carpet, cables, boxes, all the elements.
- 10:44
And then it goes on to spot the differences, and it notices, for example, that in this image, there are all these vertical lights that are not present in the first image.
- 10:52
It says that here. Uh, lighting. Like, all this, like, vertical lights on the walls and ceilings which adds... Uh, but then it rewrites the prompt, and it gets rid of all the vertical lights. [laughs] [laughs]
- 11:05
And it g- and it, and it adds the, uh, the curtain in the back, which wasn't present here but is present in the, the black curtain here. Um, so little, l- just little interesting things.
- 11:14
It's still a long way to go, but, like, uh, this, this new, this whole new... This opens a whole new box of interaction patterns, the, the, the fact that now you can reason visually.
- 11:22
Um, cool. And, and let's give a shot to, uh, a live example. So this, uh, this was a, a trail run that I did over the weekend, uh, up in [REDACTED:location].
- 11:34
Um, and so I'm just gonna do it from scratch. Um,
- 11:41
hope that it works. [laughs]
- 11:45
If you wanna go to another-
- 11:46
There you go. [laughs] Uh, cool. So the image depicts a, a serene and picturesque woodland setting. Uh, the focus of the image is a wooden boardwalk or a footbridge that winds through the dense forest.
- 11:59
Uh, very detailed description. Light filters through the trees. [laughs] Uh, and I'm just passing that raw just straight to DALL·E.
- 12:10
Yeah. And if, if folks have seen what happens in the, the DALL·E, uh, mode in the ChatGPT iOS app, for example, it's actually doing a little bit. I, I don't, uh, know off the top of my head, like, what the prompt is for that, but it's, it's doing some amount of prompt engineering.
- 12:25
Like, if folks have actually tried to use, like, our Labs product before to make DALL·E images, you have to do that prompt engineering yourself. Um, and I think that's been one of the limitations.
- 12:34
Like, if people used Midjourney or, or other, um- Other image models in the past, like it's just kind of hard to make good prompts that work well for these systems, so it's nice that the, uh, the model can, can take a stab at doing it for you.
- 12:51
It's telling us a lot of how the, the second image is a lot more beautiful and more detailed [laughs] [laughs] Which checks out. [applause]
- 13:10
It's, it's also interesting to see, uh, just for folks to, to think about. It's interesting to see that like it's still of these, um, image models, like the main limitation as we're seeing-
- 13:20
I know
- 13:20
... this demo in real time is actually- [laughs]
- 13:24
No. [laughs] [laughs] Of course. [laughs] [applause] Here you go.
- 13:33
We're going back to the slides. Next... Go back a slide to where we have, you know, the time.
- 13:37
All right. I'm gonna leave it running, and then at, at the time if we, if we have time it'll probably work the second round because-
- 13:43
It worked the three times before this [laughs]
- 13:46
Um, cool. Okay. For the second demo, um, uh, we're gonna take it a little bit further, and we're gonna do something, uh, w- with video. Uh, and the idea here is that there's a lot of video summarization demos out there that we've seen.
- 14:06
Uh, the majority of them just take a transcript and then, uh, ask GPT-4 to summarize this transcript. However, videos have a lot of, uh, information in them that is conveyed visually.
- 14:16
And so, uh, what we're doing here is that we're taking frames from the video, um, and then we're asking GPT-4 with Vision to describe all the frames, and then we are asking Whisper to transcribe the video.
- 14:29
And now we have this long textual representation of the video that not only includes all the audio information, but also includes visual information from the video. And then we're doing some exciting like mixes on that, uh, that Logan will tell you about.
- 14:42
Yeah. I'm ready for the next slide. Um, yeah, so for, for this demo we're literally just taking the GPT-4 introduction, uh, video folks have seen on YouTube. It's a good video if you haven't seen it before.
- 14:55
Um, so taking the video raw from YouTube. Go to the next slide. Uh, taking the video raw from YouTube, again, like Simón said, cutting up those, uh, the different frames from the video, putting those into, to GPT-4 with image input, getting the summaries which you can see, and I know it's really hard.
- 15:12
Um, but literally just like actually saying what's... Th- these are simple images, so it's easy to capture the, the depth of what's shown here. Um, taking those images and then going to the next piece which is essentially a big, another, another wonderful J- DALL·E image, but a big description of, uh, of the transcript and then all of
- 15:34
the image... Essentially like image embeddings is the, is the easiest way of thinking about it. So if you wanna actually see the results of this, QR code bottom right-hand corner is real.
- 15:44
Um, you can scan it and see the resulting article. It's, it's pretty- [laughs] [laughs]
- 15:52
It's pretty good. Um, it does a good job, and I think for, for me, you know, why this is exciting is 'cause you can sort of capture the, again, capture the depth of, uh, of what happens in a video.
- 16:04
So a DALL·E image to start, and then a bunch of actual frames that like match up with the contextual representation of what's being talked about in the blog post.
- 16:13
Um, and again, there's no hand... I, I couldn't open source the code 'cause it has a bunch of unreleased APIs, but no, no sort of magic behind the scenes stuff that's happening.
- 16:24
This is like a raw, crappy prompt, um, to generate this, uh, this blog post. Which I think is, again, I think it's really cool and, um, takes videos and, and makes them more accessible in, in the, in the text form.
- 16:38
So I like it.
- 16:39
Cool. Let's see if this finished. No.
- 16:46
Oh. [laughs]
- 16:47
Oh, well. Um... [laughs] Cool.
- 16:55
Can you bring it back a sec?
- 16:55
Oh, yeah. Sure. Cool. [laughs] [laughs] Okay. Uh, so some, some, uh, concluding takeaways.
- 17:11
Um, uh, start thinking multimodal. Uh, that's, that's something net new that's, that's happening these days and, and if you have any crazy ideas that you think, "Wow, it would be really cool if, if technology could do this," uh, we'll probably be able to get there and, and the products that you'll be able to build six months from
- 17:30
now, a year from now are gonna be incredible. So start having this in mind as, as, as people who are building AI products and people who are building companies.
- 17:39
Um, think of text as a, as a connecting tissue right now. Uh, and, and I think this is a very powerful concept, and that's gonna continue to be the case for the near future.
- 17:49
Uh, uh, and there are many powerful patterns that are yet to be explored when it comes to multimodal stuff, especially when it comes to, to, uh, doing things with images.
- 17:57
Uh, so really excited to, uh, soon get this in the hands of all of you guys and, and to see what you all build with this. I think it's, uh, it's really exciting, uh, to see, uh, AI start to venture into the visual world.
- 18:12
Yeah, agents with image input is gonna be sick. I can't wait.
- 18:15
Yeah.
- 18:15
I feel like so much of the internet is, requires that. Yeah, and we're excited. I think there's, there's a lot of stuff that's gonna happen in the, in the near future and, um, I think it's cool to be able to hopefully get a glimpse of, of what some of those use cases look like.
- 18:27
So anything else you wanna say, Simón?
- 18:30
No, that's good.
- 18:31
All right.
- 18:31
Yeah.
- 18:31
This was wonderful. Thank you all. [applause]
- 18:33
Thank you all. [upbeat music]