# See, Hear, Speak, Draw

Logan Kilpatrick, OpenAI · Simón Fishman, OpenAI

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/bNZV9s3_u44-see-hear-speak-draw) · [Video](https://www.youtube.com/watch?v=bNZV9s3_u44)

## See, Hear, Speak, Draw: Connecting Multimodal Models Through Text

An image reconstruction loop and a video-to-article pipeline show how text can connect visual understanding, speech transcription, and image generation.

Basic familiarity with language-model prompts and passing one model’s output into another is helpful.

### What can you build beyond a chatbot?

[0:00](https://www.youtube.com/watch?v=bNZV9s3_u44&t=0s)

How do you turn a research model into an application that solves a real problem? At OpenAI, Simón Fishman and Logan Kilpatrick work at that boundary: Fishman as an Applied engineer working with model users and unsolved problems, Kilpatrick in developer relations helping developers build products with the API. Their starting point is a company that both develops models and finds ways to put them to work.

Chatbots have already shown how much value a simple interface can deliver. Speaking in 2023, Kilpatrick calls it the year of chatbots and predicts that 2024 will be the year of multimodal models. The immediate examples are concrete: users are beginning to try image understanding in [ChatGPT’s image-capability rollout](https://openai.com/index/chatgpt-can-now-see-hear-and-speak/), and image generation is advancing from **DALL·E 2** to [**DALL·E 3**](https://openai.com/index/dall-e-3/). The question becomes what developers can build when their applications can work with more than typed conversation.

![Black slide with a globe surrounded by books, cameras and screens, labeled “2024” and “MULTI-MODAL.”](https://ai.engineer/api/talk-essay-media/e1646b33336ebf4c682fcd13edaa491c1239083fc54733ab4451fbf4cf24213f)

“Where are we going?” introduces 2024 as the year of multimodal.

### Text connects the model islands

[2:29](https://www.youtube.com/watch?v=bNZV9s3_u44&t=149s)

The architecture starts with separate models, each handling a different conversion.

| Model | Input | Output used by the application |
| --- | --- | --- |
| DALL·E | Text | Generated image |
| [**Whisper**](https://github.com/openai/whisper) | Audio | Transcript |
| [**GPT-4 with Vision**](https://openai.com/index/gpt-4v-system-card/), also called GPT-V | Images and text | Text describing or reasoning about the images |

**Text is the shared interface.** A transcript can become material for another model to reason over; an image description can become a generation prompt. The models need not share a native representation for the application to connect them.

Fishman distinguishes this composition from the future he wants: a model that can consume and produce more modalities while reasoning across them together. In the architecture demonstrated here, developers still do the wiring. Kilpatrick says that connecting modalities accounted for much of the work of building the demos; a unified model accepting text, video, or speech could remove some of that integration work. The two examples that follow explore what developers can assemble with separate models while thinking ahead to products they might build over the next six months or year. That horizon is an invitation to plan, not a release commitment.

### Describe, generate, compare, revise

[5:11](https://www.youtube.com/watch?v=bNZV9s3_u44&t=311s)

The first demonstration is a DALL·E Vision loop. Start with a real photograph, ask GPT-4 with Vision for a readable description, then send that description to DALL·E 3 to generate a synthetic version. The pipeline takes enough time that the speakers bring prepared examples and also attempt a live run.

Fishman’s first photograph comes from the hotel lobby: a display of Halloween-themed painted ladies. Vision describes the scene and produces a prompt for DALL·E, picking up details including `RIP` on a tombstone and a sign welcoming all dogs. The generated image captures parts of the scene but introduces conspicuous differences: black marble and white spiders.

The application then closes the loop:

1. Give GPT-4 with Vision both the original photograph and the generated image.
2. Ask it to identify differences between them.
3. Use those differences to produce revised generation instructions.
4. Generate another image with DALL·E.

The next image removes the black marble and makes the spider black, although it also makes the spider larger. The result is still imperfect, but it demonstrates an operation beyond describing an isolated picture: **compare a current result with a target, then turn the discrepancy into instructions.**

### Make visual evaluation part of the application

[7:38](https://www.youtube.com/watch?v=bNZV9s3_u44&t=458s)

This loop resembles work a human would otherwise perform: inspect a visual output, compare it with a reference, revise the instructions, and run the model again. Fishman acknowledges that there is a long way to go, but the demonstration moves some of that evaluation into the model pipeline. Two operations matter:

- **Description:** Convert an image into text that subsequent reasoning steps can consume.
- **Comparison:** Examine the difference between a current state and a desired state, then use that difference to guide the next action.

The second operation gives the application a target to work toward rather than merely another caption to produce.

An imagined interior-design application extends the comparison pattern. Someone moving into a new room finds an Instagram image with the atmosphere they want. They could provide that reference and ask an application to search Amazon for lamps that match it. Here the desired result is not an exact reconstruction: it is a set of objects compatible with the reference’s visual style. Fishman and Kilpatrick discuss this as a product idea, not as a shopping crawler they have built.

The roughness of the lobby result also reflects a deliberately minimal implementation. Kilpatrick describes a one-hour demo with no prompt engineering; Fishman estimates roughly 50 lines of code for the image loop. Almost all completion output goes straight into the next model. They expect additional techniques to improve the results, but do not measure that improvement here.

### Remove invented lights and restore a curtain

[10:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=606s)

A second prepared example uses a photograph Fishman took backstage immediately before the talk. Vision describes monitors, boxes, and cables; DALL·E’s reconstruction includes blue carpet, cables, and boxes. But the generated room also contains vertical lights that were not in the photograph.

The comparison identifies those extra lights on the walls and ceiling. After the prompt is rewritten, the next generation removes them and adds the black curtain visible at the rear of the original scene. These are specific, observable corrections, even though the whole room is not faithfully reconstructed. The useful capability is that a visual discrepancy can become an instruction without a person first spelling out what is wrong.

![Browser demo displaying a generated equipment room with cases on patterned carpet, cables, overhead lights and a dark curtain at the back.](https://ai.engineer/api/talk-essay-media/b23d8fab25d2c616b471d7a2c8844d87731f968dd2c9b73a0ac7b80de3bc2a03)

The revised backstage image shows equipment cases, coiled cables and a dark rear curtain.

### A live woodland run exposes the limits

[11:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=682s)

For the live attempt, Fishman selects a trail-running photograph of a woodland setting. Vision describes a wooden boardwalk or footbridge winding through dense forest, with light filtering through the trees. He passes that description directly to DALL·E.

While it runs, Kilpatrick explains another useful role for language models: writing image prompts. ChatGPT’s DALL·E mode does some prompt engineering for the user, although he does not know its exact prompt. In the earlier Labs workflow, users had to do that work themselves. He points to **Midjourney** and other image models as further examples of systems for which writing effective prompts can be difficult. A language model can mediate between what a person wants and the instructions an image generator needs.

The woodland comparison describes the generated image as more beautiful and detailed. That is a different judgment from whether it matches the photograph: an attractive reconstruction can still be an inaccurate one. The live attempt then runs into trouble, and the speakers return to the slides while leaving it running. Kilpatrick reports three successful runs before the onstage attempt, but the demonstration has not yet reached a completed result.

### A video contains more than its transcript

[13:46](https://www.youtube.com/watch?v=bNZV9s3_u44&t=826s)

The second demonstration addresses a limitation of transcript-based video summaries. Sending only the spoken words to **GPT-4** leaves out information conveyed visually. Instead, the application extracts frames and asks GPT-4 with Vision to describe them, while Whisper transcribes the audio. The two streams become a long textual representation of what the video says and shows.

The worked input is OpenAI’s GPT-4 introduction video from YouTube. Its relatively simple frames are easy to describe in text. Those descriptions join the transcript as source material for an article. Kilpatrick informally calls the visual material “image embeddings,” but the operation he describes is textual frame description, not an explicit vector-embedding step.

The assembly step can remain simple. In Python, keep the transcript and frame descriptions explicitly separated so the downstream writing prompt can refer to both:

```python
def article_prompt(transcript: str, frame_descriptions: list[str]) -> str:
    frames = "\n\n".join(
        f"Frame {index}:\n{description}"
        for index, description in enumerate(frame_descriptions, start=1)
    )
    return (
        "Write an article using both the spoken content and visual details.\n"
        "Indicate which supplied frames belong beside relevant passages.\n\n"
        f"TRANSCRIPT\n{transcript}\n\n"
        f"FRAME DESCRIPTIONS\n{frames}"
    )
```

This is the text handoff: descriptions let the writing model use visual information, while references to the supplied frames let the application retain the actual pictures.

The resulting article, [Don’t forget about GPT-4](https://medium.com/around-the-prompt/dont-forget-about-gpt-4-d5ab8c9493fc), begins with a DALL·E image and includes actual video frames matched to the surrounding discussion. The output therefore offers two kinds of access to the source: prose that incorporates visual details, and images that preserve the source’s appearance.

![Browser displaying “Don’t forget about GPT-4” above an illustration of colorful balloons and a glowing circular doorway, with article text below.](https://ai.engineer/api/talk-essay-media/08ba5d3c571d12eaeb365497c30d1adc11039f963614840c03d705389296db21)

The resulting article, “Don’t forget about GPT-4,” shown with an illustrated opening image.

Kilpatrick says the article was generated with a raw prompt, without hidden manual work, and that he could not open-source the code because it used unreleased APIs. This is a 2023 demonstration, not a current API recipe: ChatGPT’s image rollout was already underway, while public GPT-4 vision and DALL·E 3 API access followed in the [November 2023 DevDay announcement](https://openai.com/index/new-models-and-developer-products-announced-at-devday/). The product goal is straightforward—make video content accessible in text without throwing away everything the viewer could see.

### Products that can work with what they see

[16:39](https://www.youtube.com/watch?v=bNZV9s3_u44&t=999s)

Before closing, Fishman checks the woodland run again. It still has not finished. The prepared image examples and the illustrated article remain the completed demonstrations; the live run does not acquire a successful ending.

Fishman’s closing advice is to start thinking about products that incorporate multiple modalities, including ideas that still feel out of reach. His six-month-to-one-year outlook is optimistic, while his immediate architectural guidance remains practical: text will continue to connect models, and image-based interaction patterns offer substantial room for exploration.

Kilpatrick ends with image-input agents. Much of the internet requires understanding what is visible, so an agent that can interpret images can engage with information unavailable through text alone. The demos provide an early vocabulary for that work: describe what is present, compare it with what is wanted, and use the resulting text to decide what comes next.

## Resources

- [Don’t forget about GPT-4](https://medium.com/around-the-prompt/dont-forget-about-gpt-4-d5ab8c9493fc) — Logan Kilpatrick’s generated Summit demo article, combining a DALL·E 3 cover with frames from its source video.
- [Whisper](https://github.com/openai/whisper) — OpenAI’s speech-recognition code, with installation and transcription examples.
- [DALL·E 3 overview](https://openai.com/index/dall-e-3/) — An overview of DALL·E 3 image generation and ChatGPT-assisted prompt refinement.
- [GPT-4V(ision) system card](https://openai.com/index/gpt-4v-system-card/) — OpenAI’s introduction to its evaluation and mitigation work for GPT-4 image inputs, with a link to the system card.
- [ChatGPT can now see, hear, and speak](https://openai.com/index/chatgpt-can-now-see-hear-and-speak/) — The September 2023 announcement of ChatGPT image and voice capabilities and their staged rollout.
- [Multimodal APIs announced at DevDay](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) — The November 2023 release announcement for GPT-4 vision inputs, DALL·E 3 image generation and text-to-speech APIs.

## Slides

### Who are we?

An Applied engineer and developer relations’ Logan Kilpatrick work with people using OpenAI models. Logan helps developers build products and services with the API.

[0:42](https://www.youtube.com/watch?v=bNZV9s3_u44&t=42s) [Product and research illustrated by technical and scientific symbols.](https://assets.ai.engineer/sites/aiecode2025/sha256/669d7bc32b7397cb72c76812f51a4da94320579005d86e94ab84841da12940dd.webp)

Who are we?

PRODUCT & RESEARCH

1%
OHC
OH

### Where are we today?

Logan Kilpatrick calls 2023 the year of chatbots, pointing to the value people have built with simple chatbots despite their rudimentary systems.

[1:09](https://www.youtube.com/watch?v=bNZV9s3_u44&t=69s) [A chatbot illustration surrounded by circuitry, labeled 2023, year of chatbots.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd7da79041e24ea41723e5a45221df15df32a1f523afe3c2a2d92752c1f66a84.webp)

Where are we today?

2023
YEAR OF CHATBOTS

Interface text:
AIE
Engineer
SUMMIT
AUTOgpt
[illegible]

### Where are we going?

The slide casts 2024 as the year of multimodal AI. The speaker points to image understanding in ChatGPT and image generation with DALL·E 3.

[2:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=125s) [A globe surrounded by media and devices, labeled 2024 year of multi-modal.](https://assets.ai.engineer/sites/aiecode2025/sha256/24afe262ebd40c829dd1c511a9f2ca4ccf19b0a394664bc1f07349df80c6e134.webp)

Where are we going?

2024
YEAR OF
MULTI-MODAL

### Islands

DALL·E generates images, Whisper transcribes audio, and GPT-4 with Vision reasons over images and text. Text connects these separate model islands.

[2:38](https://www.youtube.com/watch?v=bNZV9s3_u44&t=158s) [Stylized islands connected by bridges.](https://assets.ai.engineer/sites/aiecode2025/sha256/65618fd8a23e766c380ffb1171f3c864f089ad97338c56063148bad62a797d79.webp)

Islands

[2:52](https://www.youtube.com/watch?v=bNZV9s3_u44&t=172s) [Six illustrated islands connected by bridges on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/19cbc548e7f9c615034bd4654e02bdf5cd7321e24c840ad6cd5fd1bce86f1b15.webp)

Islands

Interface text:
AIE
FIXIE
supabase

### Unity

A central hub illustrates a hoped-for future where models consume, produce and reason across more modalities at once. The speaker stresses that this is a direction, not today’s capability.

[3:23](https://www.youtube.com/watch?v=bNZV9s3_u44&t=203s) [A central circular structure connects to a symmetrical network of lines and shapes.](https://assets.ai.engineer/sites/aiecode2025/sha256/8af7df62f31f014147aead385286044de8e4adb825ba6cbc1fbbc4ce8165922a.webp)

Unity

### Demo: Photo to DALL·E with GPT-V

The demo uses GPT-V to describe a real image, then feeds that description to DALL·E 3 to generate a synthetic version.

[5:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=322s) [A camera lens and phone display colorful landscape imagery under the demo title.](https://assets.ai.engineer/sites/aiecode2025/sha256/eaa0f4a03d6fa208f90d326b9991a38b18a6fd386d98c0afc71a705a6d6b9c3e.webp)

Demo: Photo to DALL·E with GPT-V

DALL-E
LIVE DEMO

AIE
FIXIE
supabase
AI Engineer SUMMIT

### Forest image comparison

GPT-V describes an uploaded forest bridge photo for DALL·E 3 to recreate. The demo compares the result with the original, noting more vibrant colors and lighting that seems more focused or intense.

[5:44](https://www.youtube.com/watch?v=bNZV9s3_u44&t=344s) [A Streamlit page compares an uploaded forest bridge photo with a generated forest scene and lists differences.](https://assets.ai.engineer/sites/aiecode2025/sha256/64a73ccaac99929a926c59e465e973ce2aa433a0d9225f2e93291984f1ea8460.webp)

Uploaded Image

Description

Differences
There are significant differences between the two images:
1. Color and Contrast: The second image (AI-generated) has much more vibrant colors and contrast than the first image. The greens are more saturated, the shadows are deeper, and details in the trees and foliage are more pronounced.
2. Lighting: The original image has natural lighting which seems to be a mix of sunlight filtering through the trees and shade, whereas the AI-generated image has an artistic, almost mystical quality in terms of lighting. The light seems to be more focused or intense around the pathway and surrounding trees.

Interface text:
main · Streamlit
localhost:8501
Deploy
AIE
FIXIE
supabase

### Lobby photo upload

A lobby photo of Halloween-themed miniature houses goes to GPT-4 with Vision, which describes it and prepares a description for DALL·E to generate a new image.

[5:50](https://www.youtube.com/watch?v=bNZV9s3_u44&t=350s) [A Streamlit page shows an uploaded photo of a Halloween display with miniature houses in a lobby.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce078451e0a4ce5c24c4629c5326c7fc274abbbe99e42295616e5ef30ace7109.webp)

Choose an image...
Drag and drop file here
Limit 200MB per file • JPG, PNG, JPEG
Browse files
IMG_0146.jpeg  5.0MB

All Dogs Welcome
R.I.P

Uploaded Image
Description

ALL DOGS WELCOME

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex... 
localhost:8501
Deploy

### Photo to DALL·E with GPT-V

GPT-4 with Vision describes a Halloween lobby display for DALL·E to recreate. The generated image retains the dog-welcome sign and tombstones but changes details such as the marble.

[6:35](https://www.youtube.com/watch?v=bNZV9s3_u44&t=395s) [The demo shows part of the uploaded Halloween display photo above a generated image of miniature houses and decorations.](https://assets.ai.engineer/sites/aiecode2025/sha256/72c159b1c3ba172a875721f97c4f7be2208825d1d3e55b0ff2526895607c29eb.webp)

All Dogs Welcome
R.I.P.

Uploaded Image
Description

ALL DOGS WELCOME
R.I.P.

Differences
Description

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

[6:40](https://www.youtube.com/watch?v=bNZV9s3_u44&t=400s) [Streamlit demo displaying a detailed description of a Halloween dog-house display.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bee00502a43ca2c79289294a0b9363f6b4fae20c0e05dc7f6161062dd2a8dbc.webp)

Uploaded Image

Description

The image appears to be taken inside a lobby or entrance area with an elegant marble floor and walls. The area features a display that seems to be set up for some event or season. The dominant decorations involve dog-themed pieces alongside Halloween-style elements.

On the left, a poster is affixed to the wall with the heading "All Dogs Welcome" and displays photographs of various dog houses. Alongside the poster, there is a green patch of artificial grass that has two dolls' houses or playsets styled as dog houses on it. The house on the left has a pink and white theme while the one on the right has a more traditional green and brown color scheme. Both houses are adorned with illuminated interiors, little balconies, and miniature furniture.

More towards the center and right of the image, Halloween decorations become more prominent. Two gravestone mockups with "R.I.P." written on them are placed on the green carpet. Spidery cobwebs hung around the display together with a large spider on top.

On the far right, there is a black pillar with a haunting, spooky figure draped in white cloth resembling a ghost. On top of the pillar, you can see a sign which is not very clear, but its golden plaque indicates that there is some text or tribute.

There is also ambient lighting in the background, and reflection on the marble floor, which gives the setting a more polished look.

In summary, this looks like a well-decorated interior space probably meant for an event that embraces both a dog theme and Halloween elements.

ALL DOGS WELCOME

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

### Uploaded Halloween display

GPT-4 with Vision describes the Halloween display for a DALL·E prompt. It picks up the sign and tombstones, but describes two houses where the photo shows three colorful facades.

[6:45](https://www.youtube.com/watch?v=bNZV9s3_u44&t=405s) [Photograph of colorful miniature houses, cobwebs and tombstones in a lobby, above a written description.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4c3f021390c513ff979c7e23941db3f71218fdc23ebe19d80b1d46d5edbac15.webp)

Uploaded Image

Photo text: All Dogs Welcome; R.I.P.

Description

The image appears to be taken inside a lobby or entrance area with an elegant marble floor and walls. The area features a display that seems to be set up for some event or season. The dominant decorations involve dog-themed pieces alongside Halloween-style elements.

On the left, a poster is affixed to the wall with the heading "All Dogs Welcome" and displays photographs of various dog houses. Alongside the poster, there is a green patch of artificial grass that has two dolls' houses or playsets styled as dog houses on it. The house on the left has a pink and white theme while the one on the right has a more traditional green and brown color scheme. Both houses are adorned with illuminated interiors, little balconies, and miniature furniture.

More towards the center and right of the image, Halloween decorations become more prominent. Two gravestone mockups with "R.I.P." written on them are placed on the green carpet. Spidery cobwebs hung around the display together with a large spider on top.

On the far right, there is a black pillar with a haunting, spooky figure draped in white cloth resembling a ghost. On top of the pillar, you can see a sign which is not very clear, but its golden plaque indicates that there is some text or tribute.

There is also ambient lighting in the background, and reflection on the marble floor, which gives the setting a more polished look.

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

### First generated image

The generated image retains the dog-themed poster and Halloween display, but changes details such as the marble floor. The speaker notes that several details are off.

[6:57](https://www.youtube.com/watch?v=bNZV9s3_u44&t=417s) [Generated lobby display with miniature houses, tombstones and a dog-themed poster.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8271d68872bd98a1d8f3bd9ebeebef2a010c4ef6470e786ae41db3a789189e0.webp)

cobwebs hung around the display together with a large spider on top.

On the far right, there is a black pillar with a haunting, spooky figure draped in white cloth resembling a ghost. On top of the pillar, you can see a sign which is not very clear, but its golden plaque indicates that there is some text or tribute.

There is also ambient lighting in the background, and reflection on the marble floor, which gives the setting a more polished look.

In summary, this looks like a well-decorated interior space probably meant for an event that embraces both a dog theme and Halloween elements.

ALL DOGS WELCOME
R.I.P.

Differences
Description

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

### Image comparison

GPT with Vision compares the original and generated images, listing differences in the setting and decorations. The presenters then ask for a new image based on those differences.

[7:15](https://www.youtube.com/watch?v=bNZV9s3_u44&t=435s) [Expanded comparison listing differences in layout, background, poster, decorations and floor.](https://assets.ai.engineer/sites/aiecode2025/sha256/a109cf4a22016df50352074f0c1679e89150284d58f2ab0a5bd86092e3f3f85a.webp)

Differences

There are several differences between the two images (the first being the original and the second being the AI-generated):

1. General Layout and Complexity:
◦ The original image has a simpler and more realistic setup of the scene, while the AI-generated image has a more complex and graphical appearance.
2. Image Quality:
◦ The AI-generated image has higher resolution and lighting, showcasing a more polished and enhanced visual detail.
3. Background:
◦ In the original image, the background consist of light-colored walls and a reflective surface on the left. In the AI-generated image, the background is vastly different with darker walls, wooden panels, mirrors, and sophisticated lighting in the ceiling.
4. "All Dogs Welcome" Poster:
◦ The original image contains a simple poster or banner with text and images. In the AI-generated image, the poster is completely changed to a more elaborate one with an actual image of dogs and more pronounced design and text.
5. The Dog Houses & Decorations:
◦ In the original image, there are fewer decorative elements and the dog houses are simpler in design.
◦ The AI-generated image shows several dog houses with intricate design details and additional decorative elements like tombstones, a lamp, and more details on the dog houses, as well as more Halloween decorations.
6. Floor:
◦ The floor in the original image is plain, while in the AI-generated image, it has a much more luxurious appearance, with an intricate pattern consisting of black and white marble.
7. Objects and Details:

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex...
localhost:8501
Deploy

### Generated Halloween dog lobby

After GPT with Vision compares two images, the demo generates another lobby. The black marble is gone, and the spider is larger and black.

[7:23](https://www.youtube.com/watch?v=bNZV9s3_u44&t=443s) [Generated marble hallway with small dog houses, pumpkins and a large black spider above a welcome sign.](https://assets.ai.engineer/sites/aiecode2025/sha256/250564a3bbacbf7bfb75f560e2d1206b31a8ba5571c71c509bba8add6f5146cf.webp)

whereas the second image looks more like an artificially created, idealized version of the scene.

10. The outside scene through the window:
• In the original image, there is not much visible through the windows, while in the AI-generated image you can clearly see buildings through the right window.

These differences show the added details, changes and enhancements that the AI made to create a more visually pleasing and intricate image compared to the more simple and realistic original.

Description

ALL DOGS WELCOME

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex[illegible]
localhost:8501
Deploy
Made with Streamlit

[7:49](https://www.youtube.com/watch?v=bNZV9s3_u44&t=469s) [A Streamlit page shows comparison text above a generated lobby with a large spider, doghouses, pumpkins, and an all-dogs-welcome sign.](https://assets.ai.engineer/sites/aiecode2025/sha256/723f1beea3fc6ff7d29eda4f06131410c3d7a6432dbd6ad2b017ac9b3f13f11c.webp)

whereas the second image looks more like an artificially created, idealized version of the scene.

10. The outside scene through the window:
• In the original image, there is not much visible through the windows, while in the AI-generated image you can clearly see buildings through the right window.

These differences show the added details, changes and enhancements that the AI made to create a more visually pleasing and intricate image compared to the more simple and realistic original.

Description

ALL DOGS WELCOME

AIE
AI Engineer SUMMIT
Presented by
AUTOgpt
smol ai

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex[illegible]
localhost:8501
Deploy
Made with Streamlit

[10:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=606s) [Demo page showing a generated hallway with a spider, doghouses, and a welcome sign.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d6b1120e9c2545476dee95177401f7655dd8f6b03952ac8b219837c685cf8f7.webp)

whereas the second image looks more like an artificially created, idealized version of the scene.

10. The outside scene through the window:
• In the original image, there is not much visible through the windows, while in the AI-generated image you can clearly see buildings through the right window.

These differences show the added details, changes and enhancements that the AI made to create a more visually pleasing and intricate image compared to the more simple and realistic original.

Description

ALL DOGS WELCOME

AIE
AI Engineer SUMMIT
FIXIE
supabase

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex[illegible]
localhost:8501
Deploy
Made with Streamlit

### Backstage photo input

The uploaded backstage photo, taken just before the presenter went onstage, shows equipment cases, monitors and tangled cables.

[10:19](https://www.youtube.com/watch?v=bNZV9s3_u44&t=619s) [Image upload page displaying a backstage area with equipment cases, monitors, and cables.](https://assets.ai.engineer/sites/aiecode2025/sha256/59ceddcd698a440656603bd3475ccabd212b84e55685f70ad34c6e62bcb8eb1c.webp)

Choose an image...
Drag and drop file here
Limit 200MB per file • JPG, PNG, JPEG
Browse files
IMG_0150.jpeg  5.3MB
Uploaded Image
Description

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex...
localhost:8501
Deploy

### Backstage photo description

GPT with Vision describes the speaker’s backstage photo, identifying equipment cases, monitors and tangled cables.

[10:27](https://www.youtube.com/watch?v=bNZV9s3_u44&t=627s) [The backstage photo above an expanded description of the room, equipment, and cables.](https://assets.ai.engineer/sites/aiecode2025/sha256/70991045cacd619f0b16266a3faa247324fe91a1931248a5bcdafa3b1c3c05fa.webp)

Uploaded Image

Description

This image is of a backstage or technical area of an event setup. The room has a modern interior with walls painted in a light brown color and covered with a black curtain on the left-hand side. The floor has a patterned blue carpet.

The focus of the image is the collection of technical equipment and the related mess of cabling. In the forefront of the image, there is a camera on a tripod facing towards the left side.

A heap of various cables, mainly black, accumulate in the center-right area of the image; they are displayed in a tangled manner in front of an elongated table. On the table, several devices are visible including at least two laptops, a TV/monitor screwed on a stand, and assorted small devices which could be modems or switches.

In the middle of the image, there are stacked cases made of wood and plastic. The one on the top right appears to be a portable rackmount case. The one in the front-center has an "MVS" logo printed on it. The cases are probably used to transport, store, and protect equipment.

On the left, behind the camera, there appears to be a stage platform elevated above the floor, [illegible]

Interface text:
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

### Generated backstage image

DALL·E 3’s generated room retains the photo’s blue carpet, cables and cases, while adding vertical wall lights absent from the original.

[10:37](https://www.youtube.com/watch?v=bNZV9s3_u44&t=637s) [A generated equipment room with racks, cameras, and coils of cable.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8092acf757ddc827361903518675be3b7faf4feca611e1964776bb711841e96.webp)

including at least two laptops, a TV/monitor screwed on a stand, and assorted small devices which could be modems or switches.

In the middle of the image, there are stacked cases made of wood and plastic. The one on the top right appears to be a portable rackmount case. The one in the front-center has an "MVS" logo printed on it. The cases are probably used to transport, store, and protect equipment.

On the left, behind the camera, there appears to be a stage platform elevated above the floor, perhaps used for a performance or presentation.

Overall, the image illustrates the technical preparation area, with equipment and cables, behind the scenes at an event.

Differences
Description

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. [illegible]
localhost:8501
Deploy

### Backstage image generation and comparison

The demo compares a generated equipment room with the original, noting its vertical wall lights. A rewritten prompt produces another image without those lights and adds a black curtain.

[10:53](https://www.youtube.com/watch?v=bNZV9s3_u44&t=653s) [The generated equipment room above an expanded list comparing it with the original photo.](https://assets.ai.engineer/sites/aiecode2025/sha256/aeba37f1b4389452f92fce01ebf0f6353cd0cd837d73d1dfa36c759b7dcb6551.webp)

Differences

There are many differences between the first image, which is an original image, and the second image, which is AI-generated:

1. Room Style & Size:
- In the first image, the room appears smaller and looks like an unorganized storage or a temporary setup for an event.
- In the second image, the room looks more spacious, with a much more polished and professional setup, resembling a modern studio or control room.
2. Lighting:
- The first image has standard room lighting.
- The second image features an array of vertical lights on the walls and ceiling lights, which adds to the aesthetics of the room.
3. Seating:
- The first image doesn't show any seating arrangement.
- The second image shows a seating area with modern, comfortable sofas along the left wall.
4. Tables and Equipment:
- The first image shows tables with loose cables, a TV monitor, and unorganized equipment.

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy

[11:00](https://www.youtube.com/watch?v=bNZV9s3_u44&t=660s) [Streamlit displays a generated equipment room with cases, cables, and a dark curtain.](https://assets.ai.engineer/sites/aiecode2025/sha256/55d0235a8a48ce85f5537d566c520c2c11289a5c28c4b98db65a3078230ec433.webp)

Differences
Description

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex…
localhost:8501
Deploy
Made with Streamlit

### Woodland photograph

The presenter starts a live demo with a trail-run photograph of a forest footbridge, then passes its detailed description directly to DALL·E.

[11:37](https://www.youtube.com/watch?v=bNZV9s3_u44&t=697s) [Streamlit displays a photograph of a wooden footbridge through a forest.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ae391e6e3e838d2b9def0cba9b452a97454cb896850c989ae48c46822455291.webp)

Uploaded Image

Description

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex...
localhost:8501
RUNNING...
Stop
Deploy
Made with Streamlit

### Woodland image description

The demo describes the uploaded woodland photograph, then passes the raw description straight to DALL·E while the app creates an image.

[12:00](https://www.youtube.com/watch?v=bNZV9s3_u44&t=720s) [Streamlit shows a detailed description of the woodland photograph while creating an image.](https://assets.ai.engineer/sites/aiecode2025/sha256/79cf390ea402eb32c897880e895eb3f8f3a45a26b56af6d0e7f0e5ada07442b7.webp)

Uploaded Image

Description

This image depicts a serene and picturesque woodland setting. The focus of the image is a wooden boardwalk or footbridge that winds through the dense forest. The footbridge has a simple, rustic design comprised of wooden planks and rails, and appears to be well-maintained.

On the left side of the bridge, there are tall trees with deeply ridged bark, some of which might be redwoods or other conifers, indicating that this could be a temperate rainforest. The forest floor is carpeted with green foliage, and there are smaller trees and shrubs with lush, green leaves growing in abundance. Part of a tree stump can also be seen on the left, covered with foliage and other vegetation growing on and around it.

To the right side of the bridge, the land slopes downwards, leading to a small stream or brook with clear water and rocks at the bottom. The stream is lined with an assortment of rocks and fallen branches. Vegetation, including ferns and various plants, grows densely along the banks and reaches for the bridge.

The light filters through the trees, creating spots of brightness and shadow throughout the scene. The overall atmosphere is tranquil and inviting, making it an ideal spot for nature lovers, hikers, or anyone seeking some peace and quiet away from the hustle of city life.

Creating image...

Interface text:
main · Streamlit
main · Streamlit
main · Streamlit
Don't forget about GPT-4. Ex...
localhost:8501
RUNNING... Stop Deploy

### Demo: Whisper + GPT-V + GPT-4-32K + DALL-E 3

The video-to-blog demo uses GPT-4 with Vision to describe video frames and Whisper to transcribe the video, capturing visual information alongside speech.

[13:58](https://www.youtube.com/watch?v=bNZV9s3_u44&t=838s) [A neon video-to-blog illustration beneath the demo title.](https://assets.ai.engineer/sites/aiecode2025/sha256/59415168a3be2b5c0fa3c83e4b5abb3d10b32e25b605b614bd79ff7347d139d8.webp)

Demo: Whisper + GPT-V + GPT-4-32K + DALL-E 3

VIDEO to BLOG

### Demo: Blog-post generation

The demo moves from descriptions of video images and the transcript to a write_blog_post() call.

[15:24](https://www.youtube.com/watch?v=bNZV9s3_u44&t=924s) [A document illustration points to a code snippet calling write_blog_post().](https://assets.ai.engineer/sites/aiecode2025/sha256/dd704ef9ccb2cb01927f65452911e9429f4d93271297f691e92db10e431e7452.webp)

Demo: Whisper + GPT-V + GPT-4-32K + DALL-E 3

write_blog_post()

Interface text:
150
151
152
153

## Full transcript

[0:00](https://www.youtube.com/watch?v=bNZV9s3_u44&t=0s) [upbeat music] So you can think of OpenAI as a, as a product and research company.

[0:17](https://www.youtube.com/watch?v=bNZV9s3_u44&t=17s) Uh, eh, we build awesome models and then we think about what are some of the best ways to apply them to solve the biggest problems that humanity faces. And so there's this, uh, deployment pipeline.

[0:29](https://www.youtube.com/watch?v=bNZV9s3_u44&t=29s) Logan and I sit at the end of this deployment pipeline. We work with, uh, people in the real world that are using OpenAI's models. We spend our time thinking about what are some of the best ways to use our models, what are some of the hardest problems that haven't been solved yet, and how can we apply, uh,

[0:45](https://www.youtube.com/watch?v=bNZV9s3_u44&t=45s) OpenAI technology to solve these? Um, I'm on the Applied team, and I'm a engineer.

[0:51](https://www.youtube.com/watch?v=bNZV9s3_u44&t=51s) Yeah. And my name's Logan Kilpatrick, and I do developer relations stuff, so helping people build fun and exciting, uh, products and services using our, using our API.

[1:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=65s) So yeah, folks saw from the title of the talk, uh, we'll, we'll talk about multimodal stuff, but I think it's important to start off with, with where we are today.

[1:14](https://www.youtube.com/watch?v=bNZV9s3_u44&t=74s) And I think, you know, as, as we all know, people who have been building in the AI space for the last six, 12, 18 months, um, 2023 has really been the year of chatbots.

[1:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=82s) And I think it's, uh, it's been incredible to see how much people have actually been able to do, how much value you can create in the world with, like, just a simple chatbot.

[1:31](https://www.youtube.com/watch?v=bNZV9s3_u44&t=91s) Um, and it's, i- it still blows my mind to think about how rudimentary these systems are and how much more value that's gonna be created, um, in the next year, in the next decade.

[1:41](https://www.youtube.com/watch?v=bNZV9s3_u44&t=101s) Um, and that's why I'm excited for 2024, which I think is, is really gonna be the... I don't know if I can trademark this, but the year of multimodal models.

[1:49](https://www.youtube.com/watch?v=bNZV9s3_u44&t=109s) Um, it's a tongue twister, but also- [laughs] ... hopefully the domain is available, yearofmultimodals.com. Um- [laughs] No, don't, don't buy it if it's available. [laughs] [laughs] Uh, yeah, so I'm, I'm excited. We-- OpenAI has a ton of multimodal capabilities that are, that are in the works.

[2:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=125s) Um, some folks might have already tried some of these in ChatGPT in the iOS app or the, the web app today, things like, uh, vision, taking in images, describing them.

[2:13](https://www.youtube.com/watch?v=bNZV9s3_u44&t=133s) Um, we'll, we'll show that later on. Um, also the ability to generate images. We've had this historically with, with DALL·E 2, but, uh, DALL·E 3 really... If, if folks have tried it, it, it takes things to the next level, so excited to, to show some of that today as well.

[2:29](https://www.youtube.com/watch?v=bNZV9s3_u44&t=149s) Cool. So if you, if you think of, uh, the way that multimodal capabilities are working right now, eh, it's a little bit of a, of a setup of islands where we have DALL·E that takes texts and generates images.

[2:43](https://www.youtube.com/watch?v=bNZV9s3_u44&t=163s) We have Whisper that takes an audio and gen- generates text transcripts. Uh, we have a GPT-V with vision capabilities, GPT-4 with Vision capabilities that takes images and text and can reason over both at the same time.

[2:56](https://www.youtube.com/watch?v=bNZV9s3_u44&t=176s) Um, but right now these, these are all very disparate things. Uh, eh, however, you can think of text as a connective tissue between all of these models. Eh, and there's a lot of interesting things that, that, uh, we can build right now, uh, eh, using that paradigm.

[3:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=196s) But what we're actually really excited for is, uh, a future in which there's unity between all these modalities and, eh... And this is where we're going. This is not where we're today.

[3:28](https://www.youtube.com/watch?v=bNZV9s3_u44&t=208s) Uh, but you can, you can think of models in the same way that, like, GPT can consume, uh, images and text simultaneously. Uh, maybe in the future we'll consume even more modalities and we'll output even more modalities and we'll be able to reason about them in the, at the same time.

[3:41](https://www.youtube.com/watch?v=bNZV9s3_u44&t=221s) However, we're not there yet. And so, uh, today Logan and I are gonna show you just, like, some, some, uh, architecture patterns and some ways in which you can, uh, mimic this kind of situation with what we have available today, eh, and, and, and some of the patterns that you can start to think about as we move

[3:59](https://www.youtube.com/watch?v=bNZV9s3_u44&t=239s) towards this future in which models can, eh, reason way beyond text.

[4:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=245s) As Simón and I were, were making these demos today, um, waiting till the, the last minute as, as always, it, it was really interesting to see that, like, really much of the work of making multimodal systems today is, like, how do you hook everything up together and connect the different modalities?

[4:20](https://www.youtube.com/watch?v=bNZV9s3_u44&t=260s) And again, as Simón said, using text as sort of the, the bridge between different modalities. Um, but it, it's gonna be super interesting to see, like, how much developer efficiency gains there are when you no longer have to do that and you really just have, like, a single model that can do text in, text out, video at

[4:34](https://www.youtube.com/watch?v=bNZV9s3_u44&t=274s) some point, you know, speech in, speech out at some point. Um, so it'll be super cool to, to see when that's possible and, uh, make, make, making demos even, uh, even easier and simpler.

[4:45](https://www.youtube.com/watch?v=bNZV9s3_u44&t=285s) Cool. All right. Well, uh, we'll show you guys two demos today. Uh, a- and we'll talk about, like, some, some high-level ideas and some high-level concepts. Uh, and hopefully at the end of it you'll, you'll be inspired to think about, like, well, what are some of the things that, uh, maybe you're not able to build today, but

[5:00](https://www.youtube.com/watch?v=bNZV9s3_u44&t=300s) you'll be able to build, uh, six months, a year from now, eh, and how you should start thinking about your products, uh, um, uh, as they are able to incorporate more, uh, modalities.

[5:11](https://www.youtube.com/watch?v=bNZV9s3_u44&t=311s) Cool. So on to demo number one. Uh, this is a, it's a, it's a very, very simple DALL·E Vision loop. Um-

[5:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=322s) Yeah

[5:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=322s) ... do you wanna walk us through?

[5:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=322s) So yeah, [laughs] sorry. Um, excited to, to look at this demo. So Simón will, will pull up the demo, and I'll, I'll sort of just walk through. But the basic idea is let's take a real image.

[5:32](https://www.youtube.com/watch?v=bNZV9s3_u44&t=332s) Um, let's use GPT-V, um, or GPT-4 with, with image inputs to essentially create a nice human, uh, readable, understandable description of that image. Um, and then we'll put that into DALL·E 3 and actually go and generate a synthetic version of that image.

[5:48](https://www.youtube.com/watch?v=bNZV9s3_u44&t=348s) Um, so this whole pipeline takes a little bit to run because, uh, it's not a production, um, system at the moment. Uh, but the nice part is, uh, we've got a couple of examples ready and we can...

[6:01](https://www.youtube.com/watch?v=bNZV9s3_u44&t=361s) You wanna kick one off live as well? [laughs] We can-

[6:03](https://www.youtube.com/watch?v=bNZV9s3_u44&t=363s) Cool. Okay

[6:04](https://www.youtube.com/watch?v=bNZV9s3_u44&t=364s) ... let it run in the background. [laughs]

[6:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=365s) So very, very-- this is a, a fun, simple idea. But, uh, the, this is a, a photo that I took in the lobby downstairs, uh, just when you walk into the, the hotel.

[6:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=376s) Uh, there are these, uh, kind of like, uh, Halloween-themed painted ladies. Um, uh, and so what we did here is that we asked, uh, GPT-4 with Vision to describe this image in detail Uh, and then we asked it to, uh, generate, uh, a description for DALL·E to, uh, generate a new image based on this.

[6:36](https://www.youtube.com/watch?v=bNZV9s3_u44&t=396s) Um, uh, you can see it, it does an okay job. Here's a description of the image. Here's a prompt it uses. Uh, it, it picks up on a lot of details like the, the RIP in the tombstone and the all dogs, uh, welcome, uh, thing here, and then it generates a whole new image.

[6:52](https://www.youtube.com/watch?v=bNZV9s3_u44&t=412s) But there's a lot of details that are off, you know? Like, the, the, um, the marble is black and the, uh, the spiders are white. Eh, and so what we do next is that we pass the-

[7:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=425s) It's close.

[7:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=425s) Yeah, it's close enough.

[7:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=426s) It's close. [laughs]

[7:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=426s) Uh, [laughs] but we give the two images to the GPT with Vision again, um, and we ask it to compare them, uh, and see what are some of the differences.

[7:15](https://www.youtube.com/watch?v=bNZV9s3_u44&t=435s) And, uh, it, it picks up on a lot of the, the different details. And then we ask it to create a new, a new image based on these, uh, differences, and it goes ahead.

[7:25](https://www.youtube.com/watch?v=bNZV9s3_u44&t=445s) New image. You see. [laughs] [laughs] Uh, all the black marble is gone. The spider is now larger [laughs] and black. [laughs] Uh, but you know, it, it, it matches them closely. And I think this, this is just to illustrate.

[7:38](https://www.youtube.com/watch?v=bNZV9s3_u44&t=458s) I think there's a long way to go, but this is to illustrate the idea that there are plenty of tasks that we do right now in AI where we, we

[7:47](https://www.youtube.com/watch?v=bNZV9s3_u44&t=467s) need the human in the loop to be able to evaluate a visual output that a model produces, compare it with something else, then, like, iterate on the instructions, pass that again to another model.

[7:56](https://www.youtube.com/watch?v=bNZV9s3_u44&t=476s) And so th- that, that's a pipeline where we, like, thought that humans were very essential and that we're probably gonna continue to be essential for some time. And now that's something that, uh, the models can do by themselves.

[8:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=485s) Um, a- and there's a couple of, uh, of, uh, interesting, uh,

[8:09](https://www.youtube.com/watch?v=bNZV9s3_u44&t=489s) uh, patterns here, I think. I think one of them is describing images. That's powerful because now you have an image. Y- now you have text, and you can reason about that text.

[8:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=496s) You can do a lot of things with that text. But another really powerful element is, like, uh, comparing images, um, uh, and, and, and spotting differences, uh, like having, like, a final destination that you want to get to and, like, a current destination.

[8:29](https://www.youtube.com/watch?v=bNZV9s3_u44&t=509s) And, uh, a- and that pattern of comparing things, you can apply to a lot of things. So imagine, uh, talking now. We're, we're-- Logan and I were just chatting about, like, some other, like, ways that you can apply this and, and, and, uh, Logan's idea was imagine you are, uh, curating, uh, your room and you're, you just

[8:43](https://www.youtube.com/watch?v=bNZV9s3_u44&t=523s) moved to a new place. Uh, and you're on Instagram, and you find some images of, like, a vibe that you like and, and like, maybe some object. And, and then you can grab that image.

[8:52](https://www.youtube.com/watch?v=bNZV9s3_u44&t=532s) You can give that to GPT-4 with Vision, and you can tell it, like, "Okay, now, like, like, crawl through Amazon and find, like, all the lamps that match this vibe that I want for my room."

[9:01](https://www.youtube.com/watch?v=bNZV9s3_u44&t=541s) Um-

[9:02](https://www.youtube.com/watch?v=bNZV9s3_u44&t=542s) I want this so badly.

[9:03](https://www.youtube.com/watch?v=bNZV9s3_u44&t=543s) Yeah.

[9:03](https://www.youtube.com/watch?v=bNZV9s3_u44&t=543s) This would be so cool.

[9:03](https://www.youtube.com/watch?v=bNZV9s3_u44&t=543s) And, and, and so [laughs]

[9:04](https://www.youtube.com/watch?v=bNZV9s3_u44&t=544s) I can't do interior design, so it's like I, I would love to be able to just be like, "Get me all the stuff that matches this specific vibe." It's, it's a, a hard problem right now.

[9:12](https://www.youtube.com/watch?v=bNZV9s3_u44&t=552s) Yeah. Um, and a couple-

[9:14](https://www.youtube.com/watch?v=bNZV9s3_u44&t=554s) Simón, can I make one other quick comment, which is just-

[9:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=556s) For sure

[9:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=556s) ... I, I think also, you know, folks were, were laughing, you know, in, in good jest when this, when this third image came up, came up. I think it's important to know that there's, there's, like, no prompt engineering or anything like that that's happening.

[9:27](https://www.youtube.com/watch?v=bNZV9s3_u44&t=567s) This is, like, the, the rawest output that you can get. This is a, a one-hour demo version. So people can, uh, will hopefully go wild with this once it's available through the API and, like, ideally get much better results than, than we're seeing today.

[9:40](https://www.youtube.com/watch?v=bNZV9s3_u44&t=580s) Um, yeah.

[9:41](https://www.youtube.com/watch?v=bNZV9s3_u44&t=581s) Yeah.

[9:41](https://www.youtube.com/watch?v=bNZV9s3_u44&t=581s) Probably using a bunch of techniques that other people talked about at the conference so far.

[9:44](https://www.youtube.com/watch?v=bNZV9s3_u44&t=584s) Right.

[9:44](https://www.youtube.com/watch?v=bNZV9s3_u44&t=584s) So this is the, the very basic version of, of this demo.

[9:48](https://www.youtube.com/watch?v=bNZV9s3_u44&t=588s) Yeah. And, and we wanted to keep it simple and minimal just to illustrate the, the, the power of the models. This is as raw as you can get when it comes to the models.

[9:56](https://www.youtube.com/watch?v=bNZV9s3_u44&t=596s) The, like, there is almost all the completion output is going straight into the model, and, and I think there's, like, 50 lines of code. So, like, the majority of the power lifting is being done by the models here.

[10:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=606s) Um, um, another quick example that I'll show you guys, and then I'll try to do one live, uh, which will probably, uh, be tragic. But, uh, [laughs]

[10:17](https://www.youtube.com/watch?v=bNZV9s3_u44&t=617s) um, so this is, this is the backstage right here. [laughs] [laughs]

[10:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=622s) Uh, uh, I just took this photo right before walking on stage. Uh, uh, you can see that, uh, GPT with Vision does a really good job actually of describing that.

[10:30](https://www.youtube.com/watch?v=bNZV9s3_u44&t=630s) Uh, the, there's the monitors, and there's boxes, and there's cables, and there's whatnot. Uh, uh, and then this is the image that DALL·E generates, DALL·E 3. Uh, so you can see blue carpet, cables, boxes, all the elements.

[10:44](https://www.youtube.com/watch?v=bNZV9s3_u44&t=644s) And then it goes on to spot the differences, and it notices, for example, that in this image, there are all these vertical lights that are not present in the first image.

[10:52](https://www.youtube.com/watch?v=bNZV9s3_u44&t=652s) It says that here. Uh, lighting. Like, all this, like, vertical lights on the walls and ceilings which adds... Uh, but then it rewrites the prompt, and it gets rid of all the vertical lights. [laughs] [laughs]

[11:05](https://www.youtube.com/watch?v=bNZV9s3_u44&t=665s) And it g- and it, and it adds the, uh, the curtain in the back, which wasn't present here but is present in the, the black curtain here. Um, so little, l- just little interesting things.

[11:14](https://www.youtube.com/watch?v=bNZV9s3_u44&t=674s) It's still a long way to go, but, like, uh, this, this new, this whole new... This opens a whole new box of interaction patterns, the, the, the fact that now you can reason visually.

[11:22](https://www.youtube.com/watch?v=bNZV9s3_u44&t=682s) Um, cool. And, and let's give a shot to, uh, a live example. So this, uh, this was a, a trail run that I did over the weekend, uh, up in [REDACTED:location].

[11:34](https://www.youtube.com/watch?v=bNZV9s3_u44&t=694s) Um, and so I'm just gonna do it from scratch. Um,

[11:41](https://www.youtube.com/watch?v=bNZV9s3_u44&t=701s) hope that it works. [laughs]

[11:45](https://www.youtube.com/watch?v=bNZV9s3_u44&t=705s) If you wanna go to another-

[11:46](https://www.youtube.com/watch?v=bNZV9s3_u44&t=706s) There you go. [laughs] Uh, cool. So the image depicts a, a serene and picturesque woodland setting. Uh, the focus of the image is a wooden boardwalk or a footbridge that winds through the dense forest.

[11:59](https://www.youtube.com/watch?v=bNZV9s3_u44&t=719s) Uh, very detailed description. Light filters through the trees. [laughs] Uh, and I'm just passing that raw just straight to DALL·E.

[12:10](https://www.youtube.com/watch?v=bNZV9s3_u44&t=730s) Yeah. And if, if folks have seen what happens in the, the DALL·E, uh, mode in the ChatGPT iOS app, for example, it's actually doing a little bit. I, I don't, uh, know off the top of my head, like, what the prompt is for that, but it's, it's doing some amount of prompt engineering.

[12:25](https://www.youtube.com/watch?v=bNZV9s3_u44&t=745s) Like, if folks have actually tried to use, like, our Labs product before to make DALL·E images, you have to do that prompt engineering yourself. Um, and I think that's been one of the limitations.

[12:34](https://www.youtube.com/watch?v=bNZV9s3_u44&t=754s) Like, if people used Midjourney or, or other, um- Other image models in the past, like it's just kind of hard to make good prompts that work well for these systems, so it's nice that the, uh, the model can, can take a stab at doing it for you.

[12:51](https://www.youtube.com/watch?v=bNZV9s3_u44&t=771s) It's telling us a lot of how the, the second image is a lot more beautiful and more detailed [laughs] [laughs] Which checks out. [applause]

[13:10](https://www.youtube.com/watch?v=bNZV9s3_u44&t=790s) It's, it's also interesting to see, uh, just for folks to, to think about. It's interesting to see that like it's still of these, um, image models, like the main limitation as we're seeing-

[13:20](https://www.youtube.com/watch?v=bNZV9s3_u44&t=800s) I know

[13:20](https://www.youtube.com/watch?v=bNZV9s3_u44&t=800s) ... this demo in real time is actually- [laughs]

[13:24](https://www.youtube.com/watch?v=bNZV9s3_u44&t=804s) No. [laughs] [laughs] Of course. [laughs] [applause] Here you go.

[13:33](https://www.youtube.com/watch?v=bNZV9s3_u44&t=813s) We're going back to the slides. Next... Go back a slide to where we have, you know, the time.

[13:37](https://www.youtube.com/watch?v=bNZV9s3_u44&t=817s) All right. I'm gonna leave it running, and then at, at the end if we, if we have time it'll probably work the second round because-

[13:43](https://www.youtube.com/watch?v=bNZV9s3_u44&t=823s) It worked the three times before this [laughs]

[13:46](https://www.youtube.com/watch?v=bNZV9s3_u44&t=826s) Um, cool. Okay. For the second demo, um, uh, we're gonna take it a little bit further, and we're gonna do something, uh, w- with video. Uh, and the idea here is that there's a lot of video summarization demos out there that we've seen.

[14:06](https://www.youtube.com/watch?v=bNZV9s3_u44&t=846s) Uh, the majority of them just take a transcript and then, uh, ask GPT-4 to summarize this transcript. However, videos have a lot of, uh, information in them that is conveyed visually.

[14:16](https://www.youtube.com/watch?v=bNZV9s3_u44&t=856s) And so, uh, what we're doing here is that we're taking frames from the video, um, and then we're asking GPT-4 with Vision to describe all the frames, and then we are asking Whisper to transcribe the video.

[14:29](https://www.youtube.com/watch?v=bNZV9s3_u44&t=869s) And now we have this long textual representation of the video that not only includes all the audio information, but also includes visual information from the video. And then we're doing some exciting like mixes on that, uh, that Logan will tell you about.

[14:42](https://www.youtube.com/watch?v=bNZV9s3_u44&t=882s) Yeah. I'm ready for the next slide. Um, yeah, so for, for this demo we're literally just taking the GPT-4 introduction, uh, video folks have seen on YouTube. It's a good video if you haven't seen it before.

[14:55](https://www.youtube.com/watch?v=bNZV9s3_u44&t=895s) Um, so taking the video raw from YouTube. Go to the next slide. Uh, taking the video raw from YouTube, again, like Simón said, cutting up those, uh, the different frames from the video, putting those into, to GPT-4 with image input, getting the summaries which you can see, and I know it's really hard.

[15:12](https://www.youtube.com/watch?v=bNZV9s3_u44&t=912s) Um, but literally just like actually saying what's... Th- these are simple images, so it's easy to capture the, the depth of what's shown here. Um, taking those images and then going to the next piece which is essentially a big, another, another wonderful J- DALL·E image, but a big description of, uh, of the transcript and then all of

[15:34](https://www.youtube.com/watch?v=bNZV9s3_u44&t=934s) the image... Essentially like image embeddings is the, is the easiest way of thinking about it. So if you wanna actually see the results of this, QR code bottom right-hand corner is real.

[15:44](https://www.youtube.com/watch?v=bNZV9s3_u44&t=944s) Um, you can scan it and see the resulting article. It's, it's pretty- [laughs] [laughs]

[15:52](https://www.youtube.com/watch?v=bNZV9s3_u44&t=952s) It's pretty good. Um, it does a good job, and I think for, for me, you know, why this is exciting is 'cause you can sort of capture the, again, capture the depth of, uh, of what happens in a video.

[16:04](https://www.youtube.com/watch?v=bNZV9s3_u44&t=964s) So a DALL·E image to start, and then a bunch of actual frames that like match up with the contextual representation of what's being talked about in the blog post.

[16:13](https://www.youtube.com/watch?v=bNZV9s3_u44&t=973s) Um, and again, there's no hand... I, I couldn't open source the code 'cause it has a bunch of unreleased APIs, but no, no sort of magic behind the scenes stuff that's happening.

[16:24](https://www.youtube.com/watch?v=bNZV9s3_u44&t=984s) This is like a raw, crappy prompt, um, to generate this, uh, this blog post. Which I think is, again, I think it's really cool and, um, takes videos and, and makes them more accessible in, in the, in the text form.

[16:38](https://www.youtube.com/watch?v=bNZV9s3_u44&t=998s) So I like it.

[16:39](https://www.youtube.com/watch?v=bNZV9s3_u44&t=999s) Cool. Let's see if this finished. No.

[16:46](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1006s) Oh. [laughs]

[16:47](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1007s) Oh, well. Um... [laughs] Cool.

[16:55](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1015s) Can you bring it back a sec?

[16:55](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1015s) Oh, yeah. Sure. Cool. [laughs] [laughs] Okay. Uh, so some, some, uh, concluding takeaways.

[17:11](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1031s) Um, uh, start thinking multimodal. Uh, that's, that's something net new that's, that's happening these days and, and if you have any crazy ideas that you think, "Wow, it would be really cool if, if technology could do this," uh, we'll probably be able to get there and, and the products that you'll be able to build six months from

[17:30](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1050s) now, a year from now are gonna be incredible. So start having this in mind as, as, as people who are building AI products and people who are building companies.

[17:39](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1059s) Um, think of text as a, as a connecting tissue right now. Uh, and, and I think this is a very powerful concept, and that's gonna continue to be the case for the near future.

[17:49](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1069s) Uh, uh, and there are many powerful patterns that are yet to be explored when it comes to multimodal stuff, especially when it comes to, to, uh, doing things with images.

[17:57](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1077s) Uh, so really excited to, uh, soon get this in the hands of all of you guys and, and to see what you all build with this. I think it's, uh, it's really exciting, uh, to see, uh, AI start to venture into the visual world.

[18:12](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1092s) Yeah, agents with image input is gonna be sick. I can't wait.

[18:15](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1095s) Yeah.

[18:15](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1095s) I feel like so much of the internet is, requires that. Yeah, and we're excited. I think there's, there's a lot of stuff that's gonna happen in the, in the near future and, um, I think it's cool to be able to hopefully get a glimpse of, of what some of those use cases look like.

[18:27](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1107s) So anything else you wanna say, Simón?

[18:30](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1110s) No, that's good.

[18:31](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1111s) All right.

[18:31](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1111s) Yeah.

[18:31](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1111s) This was wonderful. Thank you all. [applause]

[18:33](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1113s) Thank you all. [upbeat music]
