AI Engineer World's Fair 2024
From Text to Vision to Voice: Exploring Multimodality with OpenAI
Read the talk
From Text Adventures to Multimodal Assistants
Romain Huet traces the move from text generation to interruptible voice, shared visual context, and a documentary pipeline that combines video, vision, and a custom voice.
From a talk by Romain Huet
Before you start: Basic familiarity with model APIs, React components, and CSS breakpoints will help with the implementation example.
A world generated one text response at a time
What can you build when a model’s interface is text in, text out? For the early OpenAI developer platform, that question led from coding assistance and translation to entire fictional worlds. Romain Huet, introducing himself as OpenAI’s developer experience lead and a former founder, connects that experimentation to iterative deployment: putting technology into contact with real use early and often, with developers helping discover what it can become. Huet reports three million developers building on the platform at the time of the talk.
The developer API launched in 2020, before ChatGPT, with GPT-3. Early applications included basic coding assistance, copy editing, and simple translation. AI Dungeon made the open-ended nature of generation especially tangible: a player explored a role-playing world, looked around, and received newly generated descriptions rather than choosing only among prewritten scenes. Huet recalls it as one of the platform’s most popular early uses.
In 2023, GPT-4 expanded what applications could ask of a model: more complex reasoning, more specific and creative responses, stronger coding, tool use, and data interpretation. The application moved beyond generating the next passage of text toward reasoning over a user’s situation. Huet’s consumer example is Spotify, which he describes as using the models to generate playlists from music taste and listening history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Removing the handoffs between modalities
GPT-4 also began the platform’s move into vision: applications could ask about images and photographs, not just text. Huet describes GPT-4 Turbo as bringing text and vision together in the same model, followed by GPT-4o, whose “o” stands for omni. Its intended interaction spans text, images, audio, and video in real time. The desktop demonstrations that follow show a preview of those capabilities; the May 2024 API launch exposed text and vision, with audio and video access planned initially for trusted partners.
The architectural distinction is clearest in voice. Previously, a developer assembled Whisper for transcription, GPT-4 for reasoning, and a text-to-speech model for the reply. Each handoff added latency, and converting speech into text lost some of the original context. A single model handling audio directly reduces the need for those intermediate translations.
| Voice architecture | Processing path | Interaction consequence |
|---|---|---|
| Separate models | Speech → transcription → reasoning → speech | Added handoffs and loss of audio context |
| Unified model | Audio input → multimodal model → audio output | Direct handling of spoken input and delivery |
The change concerns the conversation itself, not merely the number of services in a diagram. Tone, timing, and spoken content can participate in the same interaction.
Huet presents GPT-4o as retaining GPT-4-level reasoning while improving efficiency. OpenAI’s launch claims put GPT-4o at twice the speed, half the price, and five times the rate limits of GPT-4 Turbo. Those are launch comparisons, without a reproducible speed workload specified, rather than measurements from the stage demonstration. Huet adds that rate limits are continuing to rise; eliminating them altogether is an aspiration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A conversation you can interrupt
On the Mac, Huet opens ChatGPT with Option-Space and activates voice mode. He introduces the event and asks the assistant to greet the audience. Before the greeting finishes, he interrupts: the response needs more energy, and it should include people watching the livestream. The assistant then delivers a more animated welcome addressing both groups. The correction happens inside the spoken exchange, without waiting for a completed answer and starting over.
Next, he asks it to whisper a valuable secret for people building AI-native products. Its advice concerns responsible and ethical AI. He then requests another secret, quieter and much slower; this time the advice concerns keeping products adaptable and scalable. The useful demonstration is the control over vocal delivery: the request changes how the answer is spoken as well as what it says.
Huet describes the latency as conversational and occasionally almost too fast. That is his qualitative impression, not a timed result. Alongside expressive speech, the key interaction property is interruption: he can resume speaking before the model’s audio finishes. A voice interface that supports this lets the user redirect an answer at the moment it becomes unhelpful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a handwritten greeting to a page in a book
Huet enables the camera in the existing conversation. The whispering style persists, so he explicitly tells the assistant to speak normally again. After a brief exchange about how he looks onstage, he draws something and holds it up. Asked to infer his location and translate the writing, the assistant identifies the Golden Gate Bridge, concludes San Francisco, and translates “Bonjour, développeurs” as “Hello, developers.” The same exchange now joins spoken instructions, a rough drawing, and handwritten text.
The next object is a physical book. The assistant identifies Poor Charlie’s Almanack by Charlie Munger and describes its collection of speeches, philosophy, and ideas about investing and decision-making. Huet asks for a random number between 1 and 400; the assistant chooses 126, and he opens that page.
Asked for an overview, the assistant summarizes the page as Coco Chanel’s rule for success: identify the qualities customers want and supply them. This is the assistant’s onstage reading of the page. Huet remarks that it responded faster than he could read a line. The progression matters: recognizing a cover establishes the object, while interpreting an open page brings the conversation down to the particular content the user is holding.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diagnosing a layout through screen sharing
Huet replaces the camera feed with screen sharing while continuing to speak. He shows a travel application and switches to a mobile view whose responsiveness looks wrong. The assistant notices a cramped layout and squished images, then suggests media queries. Visual context establishes the symptom before either participant has identified the code responsible for it.
He first copies over code and requests a one-line explanation. The assistant identifies useAssistant, a React hook handling messages, file uploads, and streamed server responses. That description helps eliminate it as the likely layout target. Huet moves to Discover, and the assistant agrees that this component appears to set the layout for the items and descriptions. The diagnosis moves from rendered output to code responsibility rather than treating every nearby component as a candidate for editing.
Once Huet identifies Tailwind CSS, the assistant suggests adjusting the grid columns. Asked for the prefix for mobile screens, it answers sm:. Here the distinction in Tailwind CSS v3’s mobile-first rules matters: unprefixed utilities apply to the smallest screens, while sm: starts at 640px by default and continues upward. Huet proposes two columns on medium screens and retaining three on large screens. A minimal React implementation of that layout intent is:
jsx
export function Discover({ destinations }) {
return (
<div className="grid grid-cols-1 gap-4 md:grid-cols-2 lg:grid-cols-3">
{destinations.map((destination) => (
<article key={destination.id}>
<img
className="h-auto w-full"
src={destination.imageUrl}
alt={destination.imageAlt}
/>
<h2>{destination.name}</h2>
<p>{destination.description}</p>
</article>
))}
</div>
);
}
Here, grid-cols-1 supplies the base layout, and the larger breakpoints override the column count. This expresses the proposed behavior; the recording does not establish the complete final class list.
After the edit, Huet returns to the rendered application, and the assistant reports that the images and layout look better. The loop is now complete: observe the symptom, locate the relevant component, make the edit, and inspect the result. Huet calls this a trivial coding example, then points to a broader workflow he uses: reason aloud with ChatGPT, and ask it how to prompt Cursor to carry out the implementation. In that arrangement, conversation helps formulate the task before a second assistant performs the coding work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning, affordability, customization, and agents
The roadmap begins with textual intelligence. Multimodal interfaces do not remove the need for better reasoning underneath them. Huet compares current models to first graders who can help but still make mistakes, then imagines future models reaching the level of master’s students across disciplines such as medical research and scientific reasoning. His suggestion that models could become unrecognizable within a year is a forecast of further progress, not a capability demonstrated here.
The second priority is affordability, with two different ways to reduce the cost of useful work:
- Choose an appropriate model. Not every task needs the highest available intelligence. Huet reports that GPT-4 pricing fell 80% over a year. He does not specify the baseline model or input/output mix for that comparison. He also describes plans for models of different sizes, without announcing a release timeline.
- Move eligible work out of the interactive path. Huet says the Batch API, launched a couple of months before the talk, supports asynchronous workloads such as analyzing documents, photos, and images. Huet describes a further 50% pricing discount for asynchronous batch processing. The practical tradeoff is that these jobs do not need an immediate conversational response.
Third comes model customization. Huet predicts that organizations will want models adapted to their own work, with options ranging from the publicly available fine-tuning API to assisted model development. Harvey, which builds software for law firms, is his example of the latter: it worked with OpenAI to customize GPT-4 around US case law. He praises the results without supplying a quantitative measure.
Fourth, the platform should enable agents that perceive and act. Building on the vision discussed at the previous November’s DevDay, Huet describes agents coordinating multiple AI systems, securely accessing a user’s data, and managing a calendar. Multimodality supplies ways to perceive the world in which those actions occur. Devin, from Cognition Labs, is the software-development example: breaking down complex tasks, browsing documentation, and submitting pull requests. Huet then attributes to Paul Graham the observation that 22-year-old programmers can match or exceed 28-year-olds, connecting that possibility to access to AI tools. It is an attributed observation about changing developer capabilities, not a measured age comparison presented in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Six video frames become a documentary script
The final demonstration moves from using ChatGPT to composing a multimodal application. Huet introduces Sora as a preview diffusion model that generates video from text. Before the live session, he has submitted a detailed prompt for a tree-frog documentary. He plays the resulting clip, then turns to the work needed to give it narration.
The video-to-script operation has three concrete steps:
- Extract six frames from the generated video.
- Send those images to GPT-4o with vision, together with a prompt asking it to narrate what it sees.
- Generate the script by selecting
Analyze and Narrate.
This demonstration analyzes sampled images rather than making a direct video-input call. The frames provide visual evidence, while the prompt sets the desired narrative role. Huet generates the script live and says each run yields a new story, which he discovers along with the audience. The Sora clip is already prepared; the narration text is produced during the demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Giving the documentary a voice
The remaining ingredient is speech. Voice Engine creates a custom voice from a short recording. At the time of the talk, it remains a restricted preview because of safety considerations. Huet says OpenAI also uses it internally with actors to create voices heard in the API and ChatGPT.
Huet records a brief sample of himself speaking about being onstage at the event, then plays it back. He sends that recording together with the newly generated documentary script to Voice Engine. The recording supplies the voice reference; the script supplies what the synthesized voice should say. The demonstration does not state an exact required duration for his sample.
The resulting English narration describes a vibrant green frog moving along a moss-covered branch, with black and yellow markings against the surrounding foliage. Huet then demonstrates a French version and says it sounds like him speaking French, followed by a Japanese version. The pipeline has carried visual content into a script and then into multilingual speech using his voice reference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build more with the combined capabilities
The documentary joins Sora, GPT-4o vision, and the still-unreleased Voice Engine in one application. Huet returns to the four platform priorities behind that example: stronger textual intelligence, faster and more affordable models, customization, and multimodal agents. Their shared purpose is to make more applications feasible. As he puts it, the goal is for developers to “build more with OpenAI,” rather than simply spend more.
He closes by treating this as an early stage of a fundamental change in how software is built. Supporting developers and startups includes hearing what the platform still needs to do better, and he invites that feedback after the talk. The invitation to reinvent software 2.0 follows directly from the demonstrations: a conversation can be interrupted, an assistant can share the user’s visual context, and an application can pass useful content between modalities to produce something none of those steps delivers alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original announcement explains GPT-4o's unified modalities, launch capabilities, and API efficiency claims.
The 2024 preview describes custom voices from short recordings, multilingual applications, and consent safeguards.
A case study of Harvey and OpenAI collaborating on domain-specific model training for legal work.
Cognition's initial presentation of an agent that plans software tasks and uses developer tools.
The original announcement of OpenAI's general-purpose text API and request-access rollout.
Further reading
Examples explain mobile-first utilities, breakpoint prefixes, and responsive layout changes.
Explains Sora's diffusion-transformer architecture and representation of video as spacetime patches.
Updates since the talk
Current instructions for submitting asynchronous batches, retrieving results, and handling expiration.
Read the complete timestamped transcript
- 0:00
[upbeat music] Wow. Good afternoon, everyone.
- 0:16
Super excited to see you all here today. Such an incredible energy here at the event. I'm Romain. I lead developer experience at OpenAI, and before joining OpenAI, I was also a founder.
- 0:27
And like many of you in this room, I actually experienced firsthand the magic of building with the frontier models. Now, I'm working on making sure we offer the most delightful experience for all of you builders in this room.
- 0:39
And what I love the most about this role is also showing, uh, the art of the possible with our AI models and technologies. And so, uh, today we're gonna go through a few things, uh, that, uh, that the great capabilities that the OpenAI team has built recently, and we'll show some live demos to really bring them to
- 0:56
life. But first, I'd like to start with a quick, um, zoom out on how we've gotten to where we are today.
- 1:05
OpenAI is a research company, and we're working on building AGI in a way that benefits all of humanity. And to achieve this mission, we believe in iterative deployment. We really wanna make the technology enter contact with reality as early and often as possible.
- 1:19
And for that, a top focus for us at OpenAI is really all of you, like the best builders in the world. We really believe strongly that the best developers and startups are integral to the G in this like AGI mission.
- 1:31
You guys are the ones that are gonna build the native AI products in the future. So today, we have three million developers around the world building on the, uh, OpenAI platform, and we are very fortunate, uh, to, to have so much innovation.
- 1:45
But I'd like to rewind a little bit and, you know, today outside of this room, when people think of OpenAI, they often think of ChatGPT first because that's become like the, the product that has taken the world by storm a little bit.
- 1:57
But the first product was actually not ChatGPT. The first product we put out there was the developer platform. So back in twenty-twenty, at the time, we had GPT-3, and that's when we first started launching it to the public behind an API.
- 2:12
Uh, maybe a quick show of hands actually, who in this room have played, uh, with the API at the time of GPT-3 already?
- 2:19
Wow. [chuckles] That's like more than half of you. You, you guys are really my crowd here. Uh, that's awesome. And, you know, at the time, we kinda got a taste for what AI would be capable of doing, like basic coding assistance, copy editing, maybe some very simple translation.
- 2:35
But to really put things in perspective, at the time, this was the most or one of the most popular use cases on the platform, AI Dungeon. This was like a role-playing game purely based on text, and it kind of was generating open-ended stories, and you could navigate, uh, the world and, you know, at, at each scenery when
- 2:54
you were trying to look around, it would generate new text. So that was kind of the state-of-the-art at the time. [chuckles] Obviously, in twenty-twenty-three, GPT-4 changed the game. It completely, um, changed the way we thought about AI.
- 3:06
It got better at reasoning. It get more creative, more specific. It could start being better at coding and reasoning about complex problems, and it could use tools also, interpret data, and kind of that dramatically expanded the aperture of the possibilities with the platform.
- 3:23
We've had the great fortune of working with many, many developers and companies like integrating GPT-4 in their own apps and services, and this is just one example among many.
- 3:33
Uh, Spotify, when they took, um, our models to kind of generate, uh, playlists on the fly based on your music taste and history.
- 3:42
But the one thing I wanna highlight today in this talk is that GPT-4 was also the beginning of our multimodality journey. This is the very first time where we introduced like vision capabilities, and suddenly GPT-4 could analyze and interpret data, um, such as images, photos, et cetera, not just purely text.
- 4:01
And then with GPT-4 Turbo, for the first time, we brought vision capabilities into the same model right inside so that you can have the exact same model that does the, the two modalities at the same time.
- 4:14
And of course, last month, we kinda took a step further with multimodality, and that brings us to GPT-4o. So GPT-4o is our new flagship model that, that can reason across audio, video, and text all in real time.
- 4:29
And thanks to its native image and audio capabilities, we really think that you can now build, uh, very natural human-computer interactions in a way that we've never seen before.
- 4:40
And so the reason we call it GPT-4o is because O is for omni, omnimodel, if you will. And that mean-- that's because we brought all of these modalities into one single model for you to interact with.
- 4:51
And there are like many things that are very unique and interesting about GPT-4o, but I'd like to call out a couple, uh, step function changes here. The first one is that it's really a big step towards like this like, you know, multimodality future with like ultra-fast latency.
- 5:07
For the first time, like I'm not sure if many of you have played with ChatGPT with voice mode before or if you've tried to build your own voice assistance, but you kinda had to stitch kinda three models together.
- 5:18
First, like Whisper for translation, then GPT-4 for reasoning, and then ultimately text-to-speech. But of course, that kind of added a loss of context and some latency along the way.
- 5:28
Now, we have all of that into a single model. And for all of you developers, what's really exciting too is that for the same kinda state-of-the-art reasoning capabilities that we offer with GPT-4, we also ship massive efficiency improvements, and that means that in turn, we've been able to turn those efficiency, uh, gains back to you.
- 5:48
So as a result, um, GPT-4o is twice as fast as GPT-4 Turbo. It's also half the price, and we also increased drastically the rate limits. I ac- I actually have like five X higher rate limits on this slide, but we are kind of like always pushing them higher, and we really wanna live in a world where there's
- 6:05
not even any rate limits, so you can build with no limits. But with all that said, I'd like to give you a glimpse into GPT-4o, uh, and so we're gonna switch over to my Mac here.
- 6:17
So for this first live demo, I'm gonna bring up, uh, ChatGPT for desktop, and all I have to do is click Option, uh, Space, and I'm gonna bring the voice mode right here.
- 6:30
Hey, ChatGPT, how are you?
- 6:31
Hey, I'm doing great. Thanks for asking. How about you?
- 6:35
I'm great. I'm actually on stage at the AI Engineer World's Fair in San Francisco. Uh, could you greet the audience? We have some amazing developers here in the room.
- 6:44
Hey, everyone. It's awesome to be virtually here at the AI Engineer World's Fair in San Francisco. A big shout-out-
- 6:50
So, ChatGPT, I, I kinda like how you started, but I really want you to bring the energy. Like, people are very excited here, so make this more alive. And also, don't forget, people are also tuning in from the live stream, so don't forget about them.
- 7:03
Hello, San Francisco, and to everyone joining us on the live stream. Welcome to the AI Engineer World's Fair. We've got an incredible lineup and a ton of energy here today.
- 7:17
Let's dive into all the exciting tech and innovation.
- 7:21
That's much better. Thank you, ChatGPT. I think they really enjoyed that.
- 7:25
Here. If you need anything else, just let me know. Hope the event goes fantastically.
- 7:31
Yeah, maybe one thing you could do, like, since we have a crowd of the best builders out there, can you whisper one secret that's really precious but has them building the future of their AI-native products?
- 7:41
What should they know?
- 7:43
All right. Just between us, remember, the key to building great AI-native products is focusing on responsible and ethical AI.
- 7:55
Awesome.
- 7:55
Transparency and pro- Glad you liked it. Anything else I can-
- 7:58
Can you whisper another one, but even lower? Like, very, very, very slow.
- 8:06
Sure. Here's one more. It's crucial to keep your AI adaptable and scalable. Technology evolves fast. Your products should too.
- 8:18
Cool. So one-- [audience applauding] Thank you. A couple things to call out here, but first of all, like, you can see the latency is pretty im- impressive. Like, it's very much, like, natural in a conversation and, and there's literally no difference with talking to humans.
- 8:35
It's almost too fast at times, I would say. Um, the second thing that's really cool is, like, it understands emotion and can generate tones. You know, it could whisper, it could bring the energy, it could be more alive.
- 8:45
But also, I can jump back in the conversation at any time. I can interrupt the model and I don't ha- I don't have to wait for the kinda audio output to finish, and that's really cool to have something much more natural, uh, versus what we, what we knew before.
- 8:58
So the next thing I'd like to, to show you, of course, we're talking of modalities, so let's bring the second modality. I'd like to turn on camera here.
- 9:07
Anytime. Happy to be there.
- 9:09
Hey, ChatGPT, can you now see me?
- 9:11
Yes, I can see you.
- 9:14
Yeah. You know, you don't have to whisper anymore. You can talk normally now.
- 9:18
Got it. How's the event going?
- 9:20
It's good. Um, do I look good, actually? 'Cause I'm in front of a lot of people here.
- 9:26
You look great. Ready for the spotlight.
- 9:29
Awesome. Okay, so give me a second. I'm gonna try s- I'm gonna try to draw something for you and, and I'll, I'll, I'll show it to you.
- 9:37
Sounds good. I'm curious to see what you come up with.
- 9:43
Just give me a few more seconds, you know?
- 9:47
Take your time. I'm here whenever you're ready.
- 9:52
Luckily, ChatGPT's pretty patient, so that's good.
- 9:55
No rush. I'm always here when you're ready to.
- 9:58
Okay. So based on what I drew here, can you take a guess at where I am? And also, can you translate what I wrote?
- 10:06
Yes. That's the Golden Gate Bridge, so you must be in San Francisco. And you wrote, "Bonjour, developer," which means, "Hello, developers," in French.
- 10:18
Really cool. All right. Um, [audience applauding] it was a pretty poor bridge-
- 10:23
Nailed it
- 10:23
... uh, but ChatGPT is pretty clever. Um, next, I have this book. Can you take a look, ChatGPT? What is this?
- 10:31
Yeah. That's Poor Charlie's Almanack by Charlie Munger. It's a collection of his speeches, talks, and life philosophy. Munger is known for his wisdom on investing and decision-making.
- 10:43
Yeah. Can you, uh, can you just give me a random number between one and four hundred?
- 10:47
Sure. How about one hundred twenty-six?
- 10:51
One twenty-six. Here we go. Okay, page one twenty-six. Can you g- re- give me, like, a brief overview of what's happening on this page?
- 10:59
It looks like this page talks about Coco Chanel's rule for success. It mentions that Chanel focused on identifying the qualities her customers wanted and made sure to offer them.
- 11:10
Essentially, understanding your audience desires is key to success. Would you like a bit more detail from the page?
- 11:17
No, I think I'm good. This is pretty amazing. You, you read faster than I could read just one line, so it's pretty amazing. All right. Um, cool. Next, I'd like to show you something else.
- 11:25
So instead of camera, I'll go real quick here and, uh, instead of sharing camera, I'll share my screen. So I'm gonna bring up ChatGPT one more time here with audio capabilities.
- 11:35
I'm gonna reduce this window a little bit.
- 11:39
Can you clarify?
- 11:40
Okay, ChatGPT, I'm gonna share my screen with you. Can you see what I see?
- 11:46
Yes, I can see your screen. What are we looking at here?
- 11:50
Uh, well, I'm bui- I'm currently working on this app. Uh, you know, it's a travel app, and I'd like to make it a little better. Um, but there's one thing that I'm not quite sure about doing, and I'd like your help.
- 12:01
So if I go to this tab here, and if I try to put the final touches on it, feels like the responsiveness is not really good. Like, what do you think?
- 12:11
I see what you mean. Uh, when you switch to the mobile view, the layout looks a bit cramped and the images seem squished. Have you considered using media queries to adjust the layout for smaller screens?
- 12:22
It might help-
- 12:23
So let's, let's take a look at the code here. Uh, maybe you could help me. I'm gonna send you first, like, this whole component. Um, I'm gonna copy the code over to you if you can take a look.
- 12:32
Uh, can you give me like a one-line overview of what's going on in that one?
- 12:38
This code defines a React hook, useAssistant, that manages interactions with an AI assistant. It handles sending and receiving messages, file uploads, and streaming responses from a server using-
- 12:49
Okay, cool. I don't think that's where we need to change some stuff. I think it might be over here in the Discover component. What do you think?
- 12:57
Yeah, it does look like this is where the layout is being set for those items and descriptions. Um-
- 13:03
So I, I'm using Tailwind CSS. Um, what should I do exactly to kinda change it and make it responsive here?
- 13:11
You could adjust the grid color attributes on mobile sizes-
- 13:15
Okay
- 13:15
... by using mobile grid.
- 13:17
What's the, what's the kinda prefix I should use for mobile screens then?
- 13:22
You can use the SM prefix for small screens.
- 13:26
Hmm. Okay. Try that.
- 13:30
Yeah, give it a try.
- 13:31
If I say this, and maybe for, uh, other-- Should that-- Is that looking right? Like medium screen maybe two, and then we keep the three for large screens, something like that?
- 13:41
Yeah, that looks good.
- 13:43
Okay.
- 13:43
Adjusting the column-
- 13:45
Oh, take a look now. What do you think?
- 13:49
It looks much better.
- 13:51
Awesome.
- 13:51
The images and layout. Great job solving that. Anything else you wanna tweak?
- 13:59
So I really like using ChatGPT on desktop as a coding assistant, but of course, this was a very trivial use case. But what's also even more interesting is when you start reasoning out loud with ChatGPT to build something, but you also tell like, "Hey, actually I'm gonna get Cursor to do it, but what should I prompt Cursor?"
- 14:15
And I've done that many times. It's also pretty amazing to see how both of them can interact across modalities. But let's go back to my presentation, please.
- 14:24
I'd like to give you a little bit of a sneak peek of what's on our mind. What are we working on next at OpenAI as we think about these modalities and the future of models?
- 14:32
So there are four things that we're currently top of-- that are currently top of mind for us, especially for all of you developers building on the platform. The first thing is textual intelligence.
- 14:41
Of course, as you can tell, we are extremely excited about modalities, but we also think that increasing textual intelligence is still very key, uh, to unlock the transformational value of AI.
- 14:52
And we expect the potential, uh, of LLMs intelligence, that it-- we, we expect that potential to be, like, still very huge in the future. Those models today, they're pretty good, you know.
- 15:03
As we can tell, we're, we're building things with them. But at the same time, what's really cool to realize that they-- is that they, they're the dumbest they'll ever be.
- 15:10
We'll always have better models coming up. And if you will, like, it's almost like we have first graders working alongside us. They still make mistakes every now and then, but we expect that in a year from now, they might be, like, completely different and unrecognizable from what we have today.
- 15:24
They could become master students in the blink of an eye in multiple disciplines, like medical research or scientific reasoning. So we really expect the next frontier model will have such a, a function change in reasoning improvements again.
- 15:38
The second area of focus that we're excited about is, like, faster and cheaper models. And we know that not every use case requires, like, the highest intelligence. Of course, GPT-4's pricing has decreased significantly, uh, eighty percent, in fact, over a year, but we also wanna inc- in- introduce, like, more models over time.
- 15:56
So we want these models to be cheaper for you all to build. We want to have models of different sizes. We don't really have timelines to share today, but that's something we're, we're very excited about as well.
- 16:06
And finally, we wanna help you run, uh, async workloads. We launched a couple months ago the Batch API, and we're seeing, like, tremendous success already, especially for those modalities.
- 16:15
Say you have, like, documents to analyze with vision, with-- or photos or images, all that can be batched for another fifty percent, uh, discount on pricing. Third, we also believe in model customization.
- 16:27
We really believe that every company, every organization will have a customized model, and we have, like, a wide range of offering here. I'm sure many of you here have tried our fine-tuning API.
- 16:37
It's completely available for, for anyone to build with. But we also assist companies all the way, like Harvey, for instance, uh, a startup that's building a product, uh, for law firms, and they were able to kind of customize GPT-4 entirely on US case law, and they've seen, like, amazing results in doing so.
- 16:56
And last, we'll continue to invest in enabling agents. We're extremely excited about the future of agents, and we shared a little bit about that vision back in November at Dev Day.
- 17:05
And agents will be able to perceive and interact with the world using all of these modalities just like human beings. And once again, that's where the multimodality story comes into play.
- 17:15
Imagine an agent being able to kinda coordinate with multiple AI systems but also, uh, securely access your data and, and even, yes, manage your calendar and things like that.
- 17:24
We're, we're very excited about agents. Devin, of course, is an amazing example of what agents can become. Like, Cognition Labs has built this awesome, uh, uh, this aw- awesome, like, uh, software engineer that can code alongside you, but he's able to break down complex task and actually, um, you know, browse the documentation online, submit pull requests, and
- 17:46
so on and so forth. It's really a glimpse into, uh, what we can expect for the future of agents. And with all that, of that, it's no surprise that, in fact, Paul Graham realized, um, a few months ago that, like, often [REDACTED:age], uh, [REDACTED:age] [REDACTED:age], sorry, programmers are often as good, if not better, than [REDACTED:age] programmers, and
- 18:05
that's because they have these amazing AI tools at their fingertips. So with that, I'd like to switch to, uh, another demo to kinda show you this time not ChatGPT, but rather, like, what we can build with these modalities
- 18:20
So in the title of this talk, I did not mention video, but I'm sure most of you have seen Sora, the preview of our kind of diffusion model that's able to generate videos from a very simple, uh, prompt, and this is one of them.
- 18:32
So, uh, in the interest of time, I've already sent this prompt to Sora describing a documentary with a tree frog, very precise on what I'm expecting, and if I click here, this is what came out of Sora. [upbeat music]
- 18:49
It's pretty cool. But next, what I'd like to do is gonna bring this video to life. You know? And, and here what I'm doing is, like, I simply sliced frames out of the video of Sora, and what I'm gonna do next is very simple.
- 19:06
I'm gonna send these six frames over to, um, to GPT-4o with Vision with this prompt, if you're curious, and I'm gonna tell it to narrate what it sees, uh, as if it was a narrator.
- 19:18
So going back here, I'm gonna click Analyze and Narrate. Again, this is all happening in real time, so every single time the story's unique and I'm just discovering it like all of you.
- 19:28
And boom, that's it. So that's what GPT-4o with Vision was able to create based on what it saw in those frames, so it's pretty magical.
- 19:37
But last but not least, I wanted to show you one thing that we also previewed recently, uh, and it's our Voice Engine model. The Voice Engine model is the ability for us to create custom voices based on very short clips.
- 19:51
And of course, we take safety very responsibly, so this is not a model that's broadly available just yet. Uh, but I wanted to give you a sneak peek today of how it works, and also the Voice Engine is what we use internally with actors to bring the voices you know in the API or in ChatGPT.
- 20:09
So here I'm gonna go ahead and show you a quick demo. Hey, so I'm on stage at the, uh, AI Engineer World's Fair. I just, uh, need to record a few sec-seconds of my voice.
- 20:19
I'm super excited to see the audience that's really captivated by these modalities and what we can now build, uh, on the OpenAI platform. All right. Hey, so I'm on stage at the, uh- Sounds, sounds like- ...
- 20:32
AI Engineer World's Fair. I just, uh- Yeah. Sounds like it's perfect. Uh, that's all we need. So now to bring us all, bring us all together here, what I'm gonna do is I'm gonna take this clip, I'm gonna take the script that we just generated, and I'm sending all of it back to, um, the Voice Engine, and
- 20:49
we'll see what happens. [upbeat music] In the heart of the dense, misty forest, a vibrant frog makes its careful way along a moss-covered branch. Its bright green body adorned with- It's pretty cool ...
- 21:02
striking black and yellow patterns stands out amidst the lush foliage. And I can also have it translate in multiple languages, so let's try French. [upbeat music]
- 21:13
And for those who know me, that's actually how I sound when I speak French. [laughs]
- 21:26
Maybe one last one with Japanese. All right.
- 21:47
Um, thank you. Let's go back real quick to the, to the slides. [audience applauding]
- 21:55
And of course, this is one very specific examples of bringing modalities together with, you know, Sora videos, GPT-4o and Vision, the Voice Engine that we have not, uh, released yet.
- 22:06
But I hope this inspires you to see how you can kinda picture the future with these modalities combined together. So to wrap up, we're focused on these four things: textual intelligence to drive it up, uh, making our models faster and, and more affordable so you all can scale.
- 22:21
We're thinking about customizable, uh, models for your needs. And finally, making sure you can build for this multimodal future and agents. And if there's one thing I wanna leave you off with today, it's that our goal is not for you guys to spend more with OpenAI, but our goal is for you to build more with OpenAI.
- 22:39
'Cause let's remember, we're still in the very early innings of that transition, and it's a fundamental shift in how we think and build software every day, so we really wanna help you in that transition.
- 22:50
We're dedicated to supporting developers, startups. We love feedback, so z- if there's anything we could do better, please come find me after this talk. And, you know, this is really, like, the most exciting time to be building an AI-native company, so we want you to bet on the future of AI, and, and we know that bold builders
- 23:07
like all of you are gonna come up with the future and, and invent it before anyone else. So with that, thank you so much, and we can't wait to see what you're gonna build with those new modalities and reinvent software 2.0. [upbeat music]