# Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

Jyh-Jing Hwang

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/iS9YFW28XyM-waymos-emma-teaching-cars-think) · [Video](https://www.youtube.com/watch?v=iS9YFW28XyM)

## Waymo's EMMA: Teaching Cars to Think

EMMA turns camera video and routing text into future waypoints, then extends that formulation with driving rationales, multiple tasks and generated environments for evaluation.

Basic familiarity with machine-learning inputs, training targets and model evaluation is helpful; no autonomous-driving background is required.

### Why does one car drift while another navigates San Francisco?

[0:36](https://www.youtube.com/watch?v=iS9YFW28XyM&t=36s)

A research car drifts across its path in the opening driving demonstration. What separates that behavior from a vehicle that can navigate San Francisco without a driver? Jyh-Jing Hwang approaches the question through a brief history: three-layer neural networks in the 1980s, followed by increasingly deep models and end-to-end driving research. He places the illustrated wave of research around 2020, although [NVIDIA’s camera-to-steering work](https://developer.nvidia.com/blog/deep-learning-self-driving-cars/) dates to 2016. Hwang characterizes the drifting demonstration as L2-like, then contrasts it with Waymo’s L4 system.

The San Francisco demonstration shows more than street footage. Above it, a debugging view displays the system’s understanding of the scene: cars, pedestrians, cyclists, traffic lights and intersections. This explicit representation helps explain the complexity behind apparently ordinary driving.

![Slide titled “Why is an L4 system so much better?” with a vehicle visualization above street footage showing a cyclist and a stop sign.](https://ai.engineer/api/talk-essay-media/1c9405990f07523c2d9eb5321648b81c88640084f62e5a1a6e4102f0ad002483)

An L4 driving demonstration pairs a perception visualization with a street view at an intersection.

The dedicated driving stack divides the problem into three responsibilities:

| Component | Responsibility |
| --- | --- |
| Perception | Understand the current world |
| Prediction | Forecast future world states |
| Planning | Choose turns, acceleration and steering |

Perception supplies the scene that prediction projects forward; planning then decides how the vehicle should move through it. This is a complicated system, but it already solves the practical problem of driving in San Francisco.

### Geographic scale makes rare events unavoidable

[2:40](https://www.youtube.com/watch?v=iS9YFW28XyM&t=160s)

The next problem is scaling. At the time of the talk, Hwang describes rider-only service in Phoenix, San Francisco, Austin and Los Angeles, alongside a road trip visiting about ten cities that year. The expansion map reaches Tokyo, too. That destination represents exploration and development, not another rider-only service location.

A recently completed open challenge illustrates the difficulty. A marathon runner appears beside a road lined with cones. As the vehicle approaches an intersection, the traffic light is red—but a traffic controller waves it onward. The light has not changed; the relevant instruction comes from the person directing traffic. Understanding the scene requires interpreting these signals together, rather than treating the red light as an isolated observation.

**A situation that an individual driver never encounters can still recur across a large fleet.** Geographic expansion therefore demands more than recognizing familiar road objects. The system must handle unusual combinations of objects, instructions and behavior.

### Recognizing birds and a fallen scooter rider

[4:23](https://www.youtube.com/watch?v=iS9YFW28XyM&t=263s)

Foundation models offer a possible source of that broader understanding. In the first demonstration, **Gemini** receives a driving video in which a flock of birds suddenly takes flight in front of the vehicle. Recognizing birds is only part of the task: a useful driving interpretation must also anticipate how they could interact with the car. Gemini describes the event and recommends slowing down, remaining alert and adjusting speed as needed. Hwang calls this a “perfect answer” for the scenario. The displayed slide identifies the model as **Gemini 2.5 Pro**.

![Gemini 2.5 Pro slide with two residential-road images, birds visible in the upper image, and a response highlighting the flock and possible driver reactions.](https://ai.engineer/api/talk-essay-media/3c2d78cbb43d2c57b391e5894c90f757e4d4c3814e034ba934949d7ea8e45976)

Gemini describes a flock of birds taking flight and the possible need to adjust driving.

The next video shows a scooter rider slipping at night on a road wet from earlier rain. Hwang reports that Gemini identifies the incident correctly and even notices a distant gas station. These are qualitative demonstrations of scene understanding, but they motivate a concrete research question: how can that understanding become an input to driving, rather than remain a description of what happened?

### Turning video and routing text into waypoints

[6:30](https://www.youtube.com/watch?v=iS9YFW28XyM&t=390s)

[**EMMA**](https://arxiv.org/html/2410.23262v1), the End-to-End Multimodal Model for Autonomous Driving, begins with a simple research formulation built on Gemini. A router supplies high-level navigation guidance, such as turning left or right at the next intersection. That guidance becomes ordinary text. The illustrated vehicle has eight cameras covering 360 degrees; their video and the routing text enter the model together. EMMA outputs **future waypoints**: positions the vehicle should occupy over the next few seconds.

Driving logs provide the training targets. At a chosen point in a recorded drive, the model receives the available observations and routing instruction. The vehicle’s subsequent recorded positions supply the desired output. That makes the basic trajectory-learning task self-supervised: it does not require someone to label a new ideal trajectory for every training example.

The essential target construction can be expressed by splitting a position log at the current frame. Here, `positions` contains illustrative local coordinates, and the three later positions become the future-waypoint target:

```python
positions = [
    (0.0, 0.0),
    (1.0, 0.0),
    (2.0, 0.1),
    (3.0, 0.3),
    (4.0, 0.6),
]
current_frame = 1
horizon = 3

future_waypoints = positions[
    current_frame + 1 : current_frame + 1 + horizon
]
assert len(future_waypoints) == horizon
```

The important boundary is temporal: future positions are training targets, while the model’s inputs describe what is available when it must make the decision.

Two other properties distinguish this prototype:

- **Camera-only sensing:** this formulation does not use lidar; its sensory input matches the visual capabilities of the underlying model.
- **No high-definition map:** EMMA does not require the detailed map priors associated with a dedicated driving stack. It still receives ordinary routing guidance, which Hwang likens to Google Maps.

These are properties of the research model, not a description of the sensing configuration of Waymo’s deployed service.

Hwang reports state-of-the-art planning quality for the simple formulation on the nuScenes open-loop benchmark against the contemporary models in his comparison. The comparison is version-sensitive: the original EMMA paper distinguishes the base model from an internally pretrained **EMMA+**, so the reported headline should not be read as an unconditional win for every EMMA configuration.

### Producing a rationale before the trajectory

[9:17](https://www.youtube.com/watch?v=iS9YFW28XyM&t=557s)

The simple formulation leaves two opportunities unresolved. Waymo already has labeled data and expert models that could improve training, and a trajectory alone reveals little about why an end-to-end model chose it. EMMA’s next extension produces a **driving rationale** before the planned waypoints.

The reasoning sequence makes intermediate judgments visible:

1. Identify the critical road objects.
2. Describe their expected behavior.
3. Choose a high-level driving decision.
4. Output the future trajectory.

In the illustrated scene, the model identifies a cyclist and a vehicle, considers their behavior, and chooses to maintain normal speed. Other scenes can call for yielding or slowing down. The rationale provides something to inspect alongside the trajectory, although its presence alone does not establish that it faithfully explains every internal computation.

![Diagram linking camera views, a high-level command, and historical ego status to EMMA, with a driving-rationale box and a future-waypoint output.](https://ai.engineer/api/talk-essay-media/0e906ad08382e3c4f6fd93f66e278cbb87f0dc066d158687a4de4ab63df8ef37)

EMMA combines visual and text inputs with a driving rationale to plan future waypoints.

The next evaluation uses the Waymo Open Motion Dataset. Hwang describes its scale as roughly 100,000 scenarios, about 100 times the scene count of nuScenes. The specialized baselines, [**Wayformer**](https://waymo.com/research/wayformer/) and **MotionLM**, receive privileged scene information: oracle perception, a road graph serving as a high-definition map, and traffic-light states. Their task is to produce the driving output from these already structured inputs.

Hwang reports that EMMA with reasoning comes out ahead of these specialized baselines in the academic comparison he presents. That claim belongs to the presented experiment: the original paper separates its WOMD comparisons from its internal-data reasoning experiments, and its WOMD results depend on pretraining and prediction horizon. The useful architectural comparison is that a model working from visual and language inputs can compete with systems supplied with much richer structured scene information.

### More training data, lower planner perplexity

[11:39](https://www.youtube.com/watch?v=iS9YFW28XyM&t=699s)

Hwang next connects EMMA to scaling laws, crediting OpenAI with popularizing the relationship between model size, dataset size and quality. His experiment trains on a dataset substantially larger than the public academic datasets discussed earlier. Hwang reports that increasing training data continues to lower planner perplexity in the displayed experiment.

The vertical axis matters: it is **perplexity**, which Hwang interprets as a measure of planner quality, with lower values better. This supports a training-scaling trend; it does not supply a numerical conversion into collision rates or safety in interactive driving. Those questions require a different evaluation setup.

### One model, several driving tasks

[12:24](https://www.youtube.com/watch?v=iS9YFW28XyM&t=744s)

A language interface also makes the task itself flexible. Instead of training EMMA only to produce a trajectory, the researchers add tasks selected through prompts:

| Task | Requested output |
| --- | --- |
| End-to-end driving | Future vehicle trajectory |
| 3D detection | Objects in three-dimensional space |
| Road graph estimation | Road structure |
| Visual question answering | Freeform answers about the scene |

The prompt determines what EMMA should return. Its outputs are then decoded into visualizations, including trajectories, detections and road graphs. The same underlying model can therefore learn both how to move through a scene and how to describe its structure.

Hwang reports detection quality similar to other state-of-the-art models on the Waymo Open Dataset. The original paper’s detection comparison uses camera-primary LET matching with precision, recall and F1, rather than standard average precision. Hwang presents the result as evidence that different driving tasks can be co-trained, then shows their outputs together: a planned path, detected objects and an estimated road graph.

### A recorded scene cannot test every consequence

[13:57](https://www.youtube.com/watch?v=iS9YFW28XyM&t=837s)

At this point, EMMA is a research prototype with encouraging results. The next question is how to evaluate a version and establish whether its driving is safe. Hwang identifies all the preceding results as **open-loop evaluation**: replay recorded video and assess the model’s output. The vehicle’s proposed actions do not determine what the recorded camera sees next.

The evaluation methods differ in how they expose the consequences of driving decisions:

| Method | What happens |
| --- | --- |
| Open-loop replay | Assess outputs on a fixed recording |
| Simulation | Let the model drive in a virtual world |
| Real-road testing | Deploy the model in the physical world |

Hwang places greater trust in simulation than in open-loop replay, while recognizing that each method has different difficulties. A simulator can test what follows from the model’s actions; real-road testing adds the demands and risks of physical deployment.

### Changing rain and time of day

[15:04](https://www.youtube.com/watch?v=iS9YFW28XyM&t=904s)

Generative video models suggest another route into evaluation. Hwang first shows **Veo 2** from Google DeepMind as an example of realistic video generation, then introduces research on sensor simulation for end-to-end driving. He describes that work as using an open-sourced Google generative model to create videos for EMMA evaluation; he does not identify it as Veo 2. The distinction matters because the example of video realism and the actual simulation implementation are separate parts of the presentation.

The generated inputs can vary conditions such as weather and time of day. This gives the evaluator a way to ask how the planner behaves when the visual environment changes. Hwang notes that some demonstration videos were removed for time, then presents results for switching between rain and no rain and across times of day.

Hwang reports worse planner quality in rain or bad weather, attributing the degradation to changes in the camera input domain. He also reports worse quality at night than during the day, with the best performance around noon or afternoon. For a camera-only model, these experiments probe a direct dependency: the planner must derive its decisions from imagery whose visibility and appearance change with the conditions.

### Adapting foundation models for broader driving

[16:44](https://www.youtube.com/watch?v=iS9YFW28XyM&t=1004s)

The final montage shows generated road videos used to test the models, spanning different streets and lighting conditions. Hwang closes with this work still in progress. Shared Alphabet ownership gives Waymo access to Google’s foundation models, which the researchers are adapting both to interpret driving scenes and to create environments in which planners can be evaluated. The intended result is better generalization—the capability needed to extend autonomous driving beyond the places and situations already mastered.

![Four-row montage of road views showing intersections, cars, buildings, and streets under different lighting conditions.](https://ai.engineer/api/talk-essay-media/c106c1f3c68e929622742c06358023f4790076fd2153699208dffd3ebbf5f271)

A grid of driving scenes spans daylight, nighttime, and varied urban roads.

## Resources

- [Waymo introduces EMMA](https://waymo.com/blog/2024/10/introducing-emma/) — Waymo's introduction to EMMA, its shared language interface, multitask learning and deployment challenges.
- [EMMA research paper](https://arxiv.org/html/2410.23262v1) — Original formulation, benchmark tables and limitations.
- [Drive&Gen](https://arxiv.org/html/2510.06209v1) — Later publication describing controlled driving-video generation and tests of planner behavior on real versus generated inputs.
- [Wayformer](https://waymo.com/research/wayformer/) — Attention-based motion forecasting with alternative strategies for combining scene information.
- [NVIDIA's 2016 end-to-end driving research](https://developer.nvidia.com/blog/deep-learning-self-driving-cars/) — An early camera-to-steering system trained from human driving examples.

## Slides

### Autonomous driving research has a long history

The slide traces research from a three-layer network through a camera-to-steering CNN to a planning-oriented design linking tracking and mapping with motion, occupancy and a planner.

[0:28](https://www.youtube.com/watch?v=iS9YFW28XyM&t=28s) [Slide comparing early neural networks, end-to-end driving, and planning-oriented architectures.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b17f0f0fa35a7ebc88f2d4dc765a3e4d72cb505ea78e2e33e195c5618032e79.webp)

Autonomous driving research has a long history

• From a 3-layer neural network to a deep neural network.
• From a simple end-to-end model to a deliberate planning-oriented architecture.

Road Intensity Feedback Unit
45 Direction Output Units
29 Hidden Units
8x32 Range Finder Input Retina
30x32 Video Input Retina

Recorded steering wheel angle
Adjust for shift and rotation
Desired steering command
Left camera
Center camera
Right camera
Random shift and rotation
CNN
Network computed steering command
Back propagation weight adjustment
Error

Tracking
Mapping
Motion
Occupancy
Planner
(c.3) Planning-oriented Design (Ours)
perception
prediction
planning

ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK. Dean A Pomerleau. NeurIPS 1988.
End to End Learning for Self-Driving Cars. Bojarski et al. arXiv 2016.
Planning-oriented Autonomous Driving. Hu et al. CVPR 2023.

Interface text:
AIE
Microsoft
smol ai

### Autonomous driving system

Perception understands the world, prediction anticipates future states, and planning decides how to drive. The diagram places these software stages between sensor and control hardware, with offboard simulation below.

[2:04](https://www.youtube.com/watch?v=iS9YFW28XyM&t=124s) [Diagram showing sensor, perception, prediction, planning, control, and offboard simulation above a self-driving car image.](https://assets.ai.engineer/sites/aiecode2025/sha256/25f2bdf3ac25e59830e867b2a3e1147e4ad45a55125c0a5a8c01a16658a83320.webp)

Autonomous driving system

Hardware
Software

Sensor
Perception
Prediction
Planning
Control

(Offboard) Simulation

Interface text:
AIE
aws

### Where are we heading with autonomous driving?

Waymo offered rider-only service in four cities and was taking a 2025 road trip to about 10 cities, including Tokyo. Hwang says scaling brings new challenges.

[2:39](https://www.youtube.com/watch?v=iS9YFW28XyM&t=159s) [Map showing Waymo operating, upcoming and tested cities alongside 2025 road trip locations.](https://assets.ai.engineer/sites/aiecode2025/sha256/2df465bc8bcf265596db9476ab03fcd40bb1a912d1f80f02ce101ce35bbf34f0.webp)

Where are we heading with autonomous driving?

Legend: Waymo operating cities; Operation coming soon; Waymo tested cities

Seattle, WA
Upper Peninsula, MI
Upstate NY
Buffalo, NY
Boston,MA
Metropolitan Detroit Area, MI
New York, NY
Truckee, CA
San Francisco Bay Area, CA
Washington DC
Death Valley, NV
Las Vegas, NV
Los Angeles, CA
Nashville, TN
San Diego, CA
Phoenix, AZ
Atlanta, GA
Tokyo, Japan
Dallas, TX
Austin, TX
New Orleans, LA
Miami, FL

2025 Road trip cities
Las Vegas, NV
San Diego, CA
New Orleans, LA
Boston,MA
Coming soon
Coming soon
Coming soon
Coming soon

Road trip

Interface text:
AIE
Microsoft
smol ai

### Challenges in Generalization — Long-tails

A traffic controller waves a car through a red light, illustrating unusual driving situations that Hwang says recur at Waymo’s scale and must be solved to expand.

[3:56](https://www.youtube.com/watch?v=iS9YFW28XyM&t=236s) [Four video placeholders on a slide about long-tail driving challenges from the Waymo Open Dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a74567eb917ff0080ebaa992f25f441a48a5553c085c489fc3ff5ed0485cf4e.webp)

Challenges in Generalization — Long-tails

From 2025 End-to-End Driving Challenge in Waymo Open Dataset Challenges

Interface text:
AIE
aws

### Ask a multimodal large language model - Gemini 2.5 Pro

Gemini describes a flock of birds taking flight in front of the car and identifies the need for the driver to remain alert and possibly adjust speed or course.

[4:25](https://www.youtube.com/watch?v=iS9YFW28XyM&t=265s) [A driving-video prompt and Gemini response describing birds scattering in front of a vehicle, beside a black video player.](https://assets.ai.engineer/sites/aiecode2025/sha256/4673c13907e80e08b935280988e9401dfa371faa890919882a83ce2df6e9fc68.webp)

Ask a multimodal large language model - Gemini 2.5 Pro

Q: Describe this video in the context of driving?

Gemini

This video shows a driver's perspective of a residential neighborhood. The driver is navigating a paved road with houses and parked cars on either side. The camera angle suggests the driver is positioned slightly to the right of the center lane. The conditions appear sunny and clear.

As the video progresses, a large flock of birds suddenly takes flight from the ground in front of the vehicle. The birds scatter in various directions, briefly obstructing the driver's view. This unexpected event could potentially cause distraction or concern for the driver, requiring them to remain alert and possibly adjust their speed or course.

Interface text:
AIE
0:00 / 0:04
CC
Microsoft
smol ai

[4:37](https://www.youtube.com/watch?v=iS9YFW28XyM&t=277s) [A slide shows a paused video and Gemini’s description of birds taking flight in front of a vehicle.](https://assets.ai.engineer/sites/aiecode2025/sha256/c79c0169f1f6fd577a20cf8cb6e3546a2c73e81f8f37c5f20414052c09662ec1.webp)

Ask a multimodal large language model - Gemini 2.5 Pro

Q: Describe this video in the context of driving?

Gemini

This video shows a driver's perspective of a residential neighborhood. The driver is navigating a paved road with houses and parked cars on either side. The camera angle suggests the driver is positioned slightly to the right of the center lane. The conditions appear sunny and clear.

As the video progresses, a large flock of birds suddenly takes flight from the ground in front of the vehicle. The birds scatter in various directions, briefly obstructing the driver's view. This unexpected event could potentially cause distraction or concern for the driver, requiring them to remain alert and possibly adjust their speed or course.

Interface text:
AIE
Microsoft
smol ai

### Gemini response: scooter rider falls

Gemini describes a scooter rider falling near a gas station during a wet nighttime drive and advises driver caution. Hwang says it identified the scenario correctly.

[5:36](https://www.youtube.com/watch?v=iS9YFW28XyM&t=336s) [A slide shows a paused video and Gemini’s description of a scooter rider falling during a nighttime drive.](https://assets.ai.engineer/sites/aiecode2025/sha256/6094d57bea57455c967277136c5a68e88cfe17f3ff514d18d5f74e6bb1d44d7e.webp)

Ask a multimodal large language model - Gemini 2.5 Pro

Q: Describe this video in the context of driving?

Gemini

This dashcam video captures a nighttime drive on a city street. The road appears wet, possibly from recent rain, reflecting the streetlights and headlights of other vehicles.

A concerning incident occurs as a person riding an electric scooter abruptly falls off onto the sidewalk near a gas station. This unexpected event could potentially distract the driver or necessitate caution, especially if the person were to move back towards the road.

The video highlights the importance of remaining vigilant while driving, particularly at night and in potentially hazardous conditions like wet roads, as unpredictable events involving pedestrians or scooter riders can occur.

Interface text:
AIE
Microsoft
smol ai

### A research exploration: Generalizable autonomous driving

The research explores using Gemini or another multimodal LLM between sensors and vehicle control, with offboard mapping and simulation supporting the system.

[6:28](https://www.youtube.com/watch?v=iS9YFW28XyM&t=388s) [Diagram placing a multimodal LLM between sensors and control, with offboard mapping and simulation below.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ec8c42f3c59a5e38471d7a10a4904c47b47d1fa9919e223a405ef596d88fec9.webp)

A research exploration -
Generalizable autonomous driving system?

Hardware
Software

Sensor
Multimodal LLM?
Control

(Offboard) Mapping / Simulation

Interface text:
AIE
Microsoft
smol ai

### EMMA: End-to-end motion planning

EMMA combines routing commands, vehicle history and surrounding camera views to predict future waypoints. Built on Gemini, it is self-supervised, camera-only and HD map free.

[6:41](https://www.youtube.com/watch?v=iS9YFW28XyM&t=401s) [Routing text, historical context and camera views feed EMMA, built on Gemini, to produce future waypoints. The slide describes it as self-supervised, camera-only and HD map free.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d7df04b8a7e7a8761462dd9425e7937487f7fef2b7ed17acf465b40717fa86e.webp)

EMMA in the simplest form: end-to-end motion planning

Router

High-level command
turn left, turn right, go straight, ...

Context: historical ego status
anything describable in text

text
vision

EMMA: End-to-End Multimodal Model
built on top of Gemini

Planning: ego future waypoints

• Self-supervised
• Camera-only
• HD map free

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text:
AIE
Microsoft
smol ai

### State-of-the-art planner on the nuScenes benchmark

On nuScenes, EMMA+ has the lowest average L2 error in the table: 0.29 m. The benchmark predicts three seconds of driving actions from two seconds of historical data.

[8:47](https://www.youtube.com/watch?v=iS9YFW28XyM&t=527s) [The slide describes the nuScenes open-loop planning benchmark and compares methods in a table, with EMMA+ reporting an average L2 error of 0.29 meters.](https://assets.ai.engineer/sites/aiecode2025/sha256/01d54dfb51e13a2263d074ddc9a9c61f67ed03ac4885532a7b867576989ae888.webp)

State-of-the-art planner on the nuScenes benchmark

• nuScenes is one of the most popular open-loop planner benchmarks.
• 1,000 scenes, each spanning 20 seconds.
• Predict the next 3 seconds of future driving actions based on 2 seconds of historical data.

Method | self-supervised? | L2 (m) 1s | L2 (m) 2s | L2 (m) 3s | Avg L2 (m)
UniAD (Hu et al., 2023) | ✗ | 0.42 | 0.64 | 0.91 | 0.66
DriveVLM (Tian et al., 2024) | ✗ | 0.18 | 0.34 | 0.68 | 0.40
VAD (Jiang et al., 2023) | ✗ | 0.17 | 0.34 | 0.60 | 0.37
OmniDrive (Wang et al., 2024a) | ✗ | 0.14 | 0.29 | 0.55 | 0.33
Ego-MLP* (Zhai et al., 2023) | ✓ | 0.15 | 0.32 | 0.59 | 0.35
BEV-Planner (Li et al., 2024) | ✓ | 0.16 | 0.32 | 0.57 | 0.35
EMMA (random init) | ✓ | 0.15 | 0.33 | 0.63 | 0.37
EMMA | ✓ | 0.14 | 0.29 | 0.54 | 0.32
EMMA+ | ✓ | 0.13 | 0.27 | 0.48 | 0.29

High level command: go left
Legend: cluster_0, cluster_1, cluster_2, gt, selected
X-axis ticks: -250, 0, 250, 500, 750, 1000, 1250, 1500, 1750
Y-axis ticks: -200, 0, 200, 400, 600, 800, 1000
Score ticks: -34.0, -34.5, -35.0, -35.5, -36.0, -36.5, -37.0, -37.5

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text: AIE; Microsoft; smol ai

[8:52](https://www.youtube.com/watch?v=iS9YFW28XyM&t=532s) [Benchmark table compares planning errors, with EMMA+ reporting the lowest average L2 error shown.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d369f1610d068c09f52c83e162233dacd92b679f4cf21c32b6ad6f467520cb4.webp)

State-of-the-art planner on the nuScenes benchmark

• nuScenes is one of the most popular open-loop planner benchmarks.
• 1,000 scenes, each spanning 20 seconds.
• Predict the next 3 seconds of future driving actions based on 2 seconds of historical data.

Method | self-supervised? | L2 (m) 1s | L2 (m) 2s | L2 (m) 3s | Avg L2 (m)
UniAD (Hu et al., 2023) | ✗ | 0.42 | 0.64 | 0.91 | 0.66
DriveVLM (Tian et al., 2024) | ✗ | 0.18 | 0.34 | 0.68 | 0.40
VAD (Jiang et al., 2023) | ✗ | 0.17 | 0.34 | 0.60 | 0.37
OmniDrive (Wang et al., 2024a) | ✗ | 0.14 | 0.29 | 0.55 | 0.33
Ego-MLP* (Zhai et al., 2023) | ✓ | 0.15 | 0.32 | 0.59 | 0.35
BEV-Planner (Li et al., 2024) | ✓ | 0.16 | 0.32 | 0.57 | 0.35
EMMA (random init) | ✓ | 0.15 | 0.33 | 0.63 | 0.37
EMMA | ✓ | 0.14 | 0.29 | 0.54 | 0.32
EMMA+ | ✓ | 0.13 | 0.27 | 0.48 | 0.29

High level command: go left
Legend: cluster_0, cluster_1, cluster_2, gt, selected
X-axis ticks: -250, 0, 250, 500, 750, 1000, 1250, 1500, 1750
Y-axis ticks: -200, 0, 200, 400, 600, 800, 1000
Score ticks: -34.0, -34.5, -35.0, -35.5, -36.0, -36.5, -37.0, -37.5

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text: AIE; aws

### Motion planning with chain-of-thought reasoning

EMMA combines camera views with driving commands and context, then explains critical objects and their likely behavior before planning future waypoints. In this example, it identifies a cyclist and a vehicle and decides to keep speed.

[9:15](https://www.youtube.com/watch?v=iS9YFW28XyM&t=555s) [Diagram shows text and camera inputs feeding EMMA, which produces a driving rationale and future waypoints.](https://assets.ai.engineer/sites/aiecode2025/sha256/563d1308a56293bfd8f9e0c5a872bc6bf4a129fe6d57f941c5d85e24deb30764.webp)

E2E motion planning with chain-of-thought reasoning

Router

High-level command
turn left, turn right, go straight, ...

Context: historical ego status
anything describable in text

text
vision

Driving rationale
Critical objects:
cyclist at [10.13,2.46], vehicle at [8.41,-3.01]
Behavior description:
The cyclist is currently stopping at the intersection. Their anticipated trajectory indicates they might cross in front of you, potentially causing a collision if you don’t take an evasive action.
The observed vehicle is currently ahead of you, moving in the same direction, and its future trajectory suggests it will continue straight.
Meta driving decision:
Keep speed.

Chain-of-thought reasoning

EMMA: End-to-End Multimodal Model
built on top of Gemini

Planning: ego future waypoints

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text:
AIE
aws

### Waymo Open Motion Dataset benchmark

EMMA+ with chain-of-thought has the lowest L2 error at all three horizons: 0.027 m at 1s, 0.203 m at 3s and 0.543 m at 5s. Wayformer and MotionLM use inputs from oracle-quality perception or human labels.

[10:35](https://www.youtube.com/watch?v=iS9YFW28XyM&t=635s) [Table compares planning errors for MotionLM, Wayformer, and EMMA variants across prediction horizons.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc0516757a42e86c57715e5927a3900856186dc75919f27115251c60979854dc.webp)

State-of-the-art planner on Waymo Open Motion Dataset

• 103k real-world driving scenarios, 100x larger than nuScenes.
• Wayformer & MotionLM takes inputs of agent location history, agent interactions, the road graph, and traffic light state. Inputs are generated either by oracle-quality offboard perception or human labels.

Method | L2 (m) 1s | L2 (m) 3s | L2 (m) 5s
MotionLM* (Seff et al., 2023) | 0.045 | 0.266 | 0.696
Wayformer* (Nayakanti et al., 2023) | 0.046 | 0.252 | 0.628
EMMA† (based on PaLI) | 0.034 | 0.274 | 0.797
EMMA+† (based on PaLI) | 0.031 | 0.239 | 0.680
EMMA | 0.032 | 0.248 | 0.681
EMMA (w/ CoT) | 0.030 | 0.241 | 0.664
EMMA+ | 0.030 | 0.225 | 0.610
EMMA+ (w/ CoT) | 0.027 | 0.203 | 0.543

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text:
AIE
Microsoft
smol ai

### Quality keeps improving if trained with more data!

With 100% of scenarios, evaluation perplexity falls from 52.05 to 44.69 as training FLOPs increase; smaller subsets turn upward earlier. Hwang says lower perplexity means better planner results.

[11:36](https://www.youtube.com/watch?v=iS9YFW28XyM&t=696s) [Chart plots evaluation perplexity against training compute for different fractions of training scenarios.](https://assets.ai.engineer/sites/aiecode2025/sha256/37bfe781e4b9d8fcfad175271780d04a3f8ad1f422712e9578ef4961b8682266.webp)

Quality keeps improving if trained with more data!

End-to-End Motion Planning Quality Scales with Training Data

Y-axis: eval perplexity
Y-axis ticks: 44, 46, 48, 50, 52
X-axis: training flops
X-axis ticks: 10¹⁹, 10²⁰, 10²¹
Legend: 3% scenarios; 12% scenarios; 30% scenarios; 100% scenarios
Labeled points: 52.05, 50.13, 48.57, 47.56, 46.79, 46.07, 45.26, 44.69

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text: AIE; aws

[11:40](https://www.youtube.com/watch?v=iS9YFW28XyM&t=700s) [Chart showing evaluation perplexity decreasing as training compute increases, with curves for different fractions of training scenarios.](https://assets.ai.engineer/sites/aiecode2025/sha256/c059e8f9323c938390d009ddf20165d584de84f2c0b6f2b4f258f545790879ae.webp)

Quality keeps improving if trained with more data!

End-to-End Motion Planning Quality Scales with Training Data

Y-axis: eval perplexity
Y-axis ticks: 44, 46, 48, 50, 52
X-axis: training flops
X-axis ticks: 10¹⁹, 10²⁰, 10²¹
Legend: 3% scenarios; 12% scenarios; 30% scenarios; 100% scenarios
Labeled points: 52.05, 50.13, 48.57, 47.56, 46.79, 46.07, 45.26, 44.69

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text: AIE; aws

### EMMA as a generalist driving model

EMMA combines camera video with task prompts to produce driving waypoints, 3D detections, roadgraph estimates and answers about road conditions. Selected outputs are decoded into visual overlays.

[12:22](https://www.youtube.com/watch?v=iS9YFW28XyM&t=742s) [Diagram connecting driving prompts and camera input to EMMA, with answers and decoded visualizations for driving, object detection and roadgraph tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed7f80607be9ebdd803918353a3c0c9be90c07111e041cfd9296e171dead0e7c.webp)

EMMA is also a generalist driving model

Task Prompts and Context (not shown)
What’s my future driving actions?
Detect everything in 3D?
Estimate the drivable roadgraph?
Is the road ahead temporarily blocked?
[more task prompts]

Camera Input (Video)

EMMA: End-to-End Multimodal Model
built on top of Gemini

Answers
My future waypoints are x_1, y_1 and x_2, y_2 and ...
Detected objects in 3D: (x, y, z, l, w, h, theta, vehicle), ...
The lanes I can drive towards are: (x_1, y_1 and x_2, y_2, ... valid), ...
No, the road ahead is clear.
[more task answers]

Decoded Visualization

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text:
AIE
Microsoft
smol ai

### 3D object detection on the Waymo Open Dataset

EMMA+ scores 0.85 F1 on vehicles, versus 0.82 maximum F1 for both comparison models. On pedestrians, it scores 0.62 versus 0.70 and 0.63.

[13:23](https://www.youtube.com/watch?v=iS9YFW28XyM&t=803s) [Vehicle and pedestrian precision plots compare EMMA and EMMA+ with other 3D object detectors.](https://assets.ai.engineer/sites/aiecode2025/sha256/3075cafa1c410386caa3e778bc469c61e608b68bb88c2982bbd3caa65de54025.webp)

Promising 3D object detector on Waymo Open Dataset

• Without any special tweeks, EMMA maps camera images to 3D bounding boxes and achieves state-of-the-art quality!

Vehicle
Y-axis: precision
Y-axis ticks: 0.2, 0.4, 0.6, 0.8, 1.0
X-axis ticks: 0.2, 0.4, 0.6, 0.8, 1.0
F1: 0.85 EMMA+
F1: 0.61 EMMA
F1-max: 0.82 BEVFormer
F1-max: 0.82 MV-FCOS3D++

Pedestrian
Y-axis: precision
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
X-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
F1: 0.62 EMMA+
F1: 0.25 EMMA
F1-max: 0.70 BEVFormer
F1-max: 0.63 MV-FCOS3D++

EMMA: End-to-End Multimodal for Autonomous Driving. Hwang et al. Technical Report 2024.

Interface text:
AIE
aws

### A research exploration - end-to-end simulation

The diagram places a multimodal LLM between sensor and control blocks, with offboard simulation below. Hwang asks how to evaluate and validate the prototype for safety.

[13:54](https://www.youtube.com/watch?v=iS9YFW28XyM&t=834s) [Sensor, multimodal LLM and control blocks above an offboard simulation block and an autonomous vehicle image.](https://assets.ai.engineer/sites/aiecode2025/sha256/c674e460a0be0605b2297e23ae36d44562c1628e2665208a0687b4618717d3fa.webp)

A research exploration - end-to-end simulation

Sensor
Multimodal LLM?
Control

(Offboard) Simulation

Interface text:
AIE
Microsoft
smol ai

[14:00](https://www.youtube.com/watch?v=iS9YFW28XyM&t=840s) [Diagram showing sensors, a multimodal LLM and control above an offboard simulation, with an autonomous vehicle photo.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1be512e1f4d0cee09768e7061a090177c9ed5c61bd195d912b38d710769193c.webp)

A research exploration - end-to-end simulation

Sensor
Multimodal LLM?
Control

(Offboard) Simulation

Interface text:
AIE
aws

### Generative video for simulation

Hwang points to Google DeepMind’s Veo 2 as an example of realistic generative video and considers using such video for driving simulation.

[15:02](https://www.youtube.com/watch?v=iS9YFW28XyM&t=902s) [Slide asks whether realistic generative video can be used, above a black video player labeled Veo 2 from Google DeepMind.](https://assets.ai.engineer/sites/aiecode2025/sha256/561199ec7b0298fd0e076ebb475238031d1f0b469462efbb332bacd0956abe25.webp)

GenAI videos are so realistic. Can we leverage it?

Veo 2 from Google DeepMind

AIE
Microsoft
smol ai

Interface text:
0:00 / 0:12
CC

### Sensor Simulation for End-to-End Driving Evaluation

A conditioned video generator varies weather, time of day and other inputs to test an end-to-end planner. The slide also proposes using the planner to evaluate video quality and synthetic data to improve the planner.

[15:20](https://www.youtube.com/watch?v=iS9YFW28XyM&t=920s) [A conditional video-generation architecture and bullets describe evaluating video quality and planner performance using generated driving video.](https://assets.ai.engineer/sites/aiecode2025/sha256/e98af1af46c3b2a8cdba02afa42c358dd5ab6420b0d71e932ee0a76b341ecd15.webp)

Sensor Simulation for End-to-End Driving Evaluation

Pixel Space
x
Encoder
Decoder
x̃

Latent Space
z
Diffuse
zₜ
Denoiser
Transformer Block (×L)
Cross Attn
S-T Self Attn
Cross Attn
Self Attn
Inference
Denoiser × T
z̃
z_T

Condition
Bounding Box
Road Map
Ego Car Pose
Time of Day
Weather
random dropout
Encoders
c₁
c₂
AdaLN
Cross Attn

By connecting a driving video generation model with an end-to-end planner, we can:
• Evaluate video generation quality via planner
• Assess planner performance under different conditions via video generation model
• Improve planner via synthetic data from the video generation model

[illegible]rive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models. Wang et al. In submission.

Interface text:
AIE
Microsoft
smol ai

[15:28](https://www.youtube.com/watch?v=iS9YFW28XyM&t=928s) [Diagram of a conditioned driving-video generation model, with proposed uses for evaluating and improving a planner.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d9dc737c6991288633f6fcc7ebf0dabf12f4d2cc013dfc8d468c861c318c2e8.webp)

Sensor Simulation for End-to-End Driving Evaluation

Pixel Space
x
Encoder
Decoder
x̃

Latent Space
z
Diffuse
zₜ
Denoiser
Transformer Block (×L)
Cross Attn
S-T Self Attn
Cross Attn
Self Attn
Inference
Denoiser × T
z̃
z_T

Condition
Bounding Box
Road Map
Ego Car Pose
Time of Day
Weather
random dropout
Encoders
c₁
c₂
AdaLN
Cross Attn

By connecting a driving video generation model with an end-to-end planner, we can:
• Evaluate video generation quality via planner
• Assess planner performance under different conditions via video generation model
• Improve planner via synthetic data from the video generation model

[illegible]rive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models. Wang et al. In submission.

Interface text:
AIE
Microsoft
smol ai

### Test Planners in Various Conditions

EMMA’s ADE scores are higher in rain than without rain at all three horizons. Noon has the best scores at 3 and 5 seconds; 18:00 has the best score at 1 second.

[15:41](https://www.youtube.com/watch?v=iS9YFW28XyM&t=941s) [Tables compare planner ADE scores across rain conditions and times of day.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c5dbee2b20ec711f70c1e4842ceb9c09bd65e813719953d6aef6d6e996b4b45.webp)

Test planners in various conditions

Weather | ADE@1s | ADE@3s | ADE@5s
No rain | 0.0299 | 0.2853 | 0.8580
Rain | 0.0303 | 0.2910 | 0.8736
Table 3. ADE scores under varied weather. [Best, Worst].

Time-of-day | ADE@1s | ADE@3s | ADE@5s
00:00 | 0.0301 | 0.2907 | 0.8760
06:00 | 0.0301 | 0.2907 | 0.8744
12:00 | 0.0302 | 0.2886 | 0.8653
18:00 | 0.0298 | 0.2893 | 0.8747
Table 4. ADE scores under varied time. [Best, Worst].

[illegible]rive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models. Wang et al. In submission.

Interface text:
AIE
aws

### Test Planners in Various Conditions

EMMA’s ADE is higher in rain at all three horizons. Across times of day, noon has the lowest ADE at 3 and 5 seconds, while 18:00 is lowest at 1 second.

[15:55](https://www.youtube.com/watch?v=iS9YFW28XyM&t=955s) [Tables compare planner ADE scores across rain conditions and times of day.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff6ea04f10ece27e74b671f52ad159cd7b715e097c645abed89c4eb8e5c2e610.webp)

Test planners in various conditions

Weather | ADE@1s | ADE@3s | ADE@5s
No rain | 0.0299 | 0.2853 | 0.8580
Rain | 0.0303 | 0.2910 | 0.8736
Table 3. ADE scores under varied weather. [Best, Worst].
Best values are bold; worst values are underlined.

Time-of-day | ADE@1s | ADE@3s | ADE@5s
00:00 | 0.0301 | 0.2907 | 0.8760
06:00 | 0.0301 | 0.2907 | 0.8744
12:00 | 0.0302 | 0.2886 | 0.8653
18:00 | 0.0298 | 0.2893 | 0.8747
Table 4. ADE scores under varied time. [Best, Worst].
Best values are bold; worst values are underlined.

[illegible]rive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models. Wang et al. In submission.

AIE
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=iS9YFW28XyM&t=0s) [on-hold music] So I, I think a lot of people here already took Waymo here in San Francisco.

[0:21](https://www.youtube.com/watch?v=iS9YFW28XyM&t=21s) So we drive every day, and I think the technology just works as you all witness. So, uh, I'm gonna talk a little bit about the history of autonomous driving research, like what it took to, to, to take us here and in the current status.

[0:36](https://www.youtube.com/watch?v=iS9YFW28XyM&t=36s) So in the autonomous driving history, uh, people started the research in 1980s, where it's just very simple neural networks. There are three layers. And over time, people start to think about, you know, going deeper and deeper.

[0:51](https://www.youtube.com/watch?v=iS9YFW28XyM&t=51s) And then in around, uh, 2020, there is papers from NVIDIA and other, uh, research labs publishing end-to-end driving models. And this is one of the videos that they published at that time, right?

[1:04](https://www.youtube.com/watch?v=iS9YFW28XyM&t=64s) And then we can, we can check it out, right? The, the, the car in front driving is the autonomous driving cars from the research labs at that time. And you can see that it's actually drifting, right?

[1:15](https://www.youtube.com/watch?v=iS9YFW28XyM&t=75s) So this is more like a L2, uh, sort of technology, and [chuckles] you, you wouldn't want to basically ride in it. So why is an L4 system that Waymo has so much better, right?

[1:27](https://www.youtube.com/watch?v=iS9YFW28XyM&t=87s) You can see it drives around the San Francisco in various regions, in downtown areas, and it avoids pedestrians, cyclists, and it makes roads very safe. So, so the secret is very basically on the top, where, uh, you can see that this is a sort of visualization or debugging screen, where it visualize everything that the system understands.

[1:50](https://www.youtube.com/watch?v=iS9YFW28XyM&t=110s) So it can actually capture all the cars, pedestrians, cyclists, traffic lights, crossroads, and it understand almost everything related to driving. And we can see, uh, this is a very complicated system.

[2:05](https://www.youtube.com/watch?v=iS9YFW28XyM&t=125s) If we break it down, it's basically a perception system where it understands the world, and there is a prediction system that basically predicts the future of... future world states given the current one.

[2:18](https://www.youtube.com/watch?v=iS9YFW28XyM&t=138s) And finally, there is a planning system that basically tell us, like, how we should drive in this scenario. Like, we should turn right, how much acceleration, how much steering, et cetera.

[2:29](https://www.youtube.com/watch?v=iS9YFW28XyM&t=149s) So this is the, the whole autonomous driving system that is very complicated and dedicated. But it, it solves problems, right? It drives in San Francisco today. So what else?

[2:40](https://www.youtube.com/watch?v=iS9YFW28XyM&t=160s) So what are we heading with autonomous driving? The, the answer is scaling, right? So we've proven that we are operating in Phoenix, San Francisco, Austin, and Los Angeles today, and it's offering the rider-only service.

[2:57](https://www.youtube.com/watch?v=iS9YFW28XyM&t=177s) However, we are-- our ambition is not going to stop in just four cities now. So in this year, we are doing a road trip where we are visiting about 10 cities, um, in various locations.

[3:09](https://www.youtube.com/watch?v=iS9YFW28XyM&t=189s) Here is the map. And you can even see that we are actually going across the globe to, to Tokyo, Japan. So there is a lot of challenges when we start to scale.

[3:24](https://www.youtube.com/watch?v=iS9YFW28XyM&t=204s) So just recently, we finished a open challenge for participants to solve some of the hardest problems that we see in our scaling. So let's replay the video a little bit here.

[3:38](https://www.youtube.com/watch?v=iS9YFW28XyM&t=218s) So on the, on the top left, there is a marathon runner on the right, and then there is, uh, cones around the, the, the, the roads. And most interestingly is that when we drive, uh, closer to the intersection, if you look at the traffic light, it's actually red.

[3:55](https://www.youtube.com/watch?v=iS9YFW28XyM&t=235s) But there is a traffic control man waving us to go ahead. So it's actually green. [laughing] And, and this is very confusing, right? And all these long tails are very challenging to solve.

[4:06](https://www.youtube.com/watch?v=iS9YFW28XyM&t=246s) And there are challenges in, like, a lot of times that you've never seen it in your entire life of driving. But for Waymo, at Waymo scale, we will see very, very often, and we need to solve them to, in order to scale.

[4:23](https://www.youtube.com/watch?v=iS9YFW28XyM&t=263s) So one of the, the, the, the solutions arises where foundation model is, is very generalizable, right? So let's just take a example here that we feed in Gemini. Right, you can see the video where when we were driving peacefully, suddenly there are a, a, a bunch of angry birds, right?

[4:41](https://www.youtube.com/watch?v=iS9YFW28XyM&t=281s) They, they appear and attack our car virtually, right? So that's where we need to react. But for a car to understand the scenario, it also unders-- has to understand, okay, this is a flock of birds and how they are going to behave, how they are going to interact with the car, right?

[4:58](https://www.youtube.com/watch?v=iS9YFW28XyM&t=298s) This is not the everyday driving that you see, but this is a critical long tail. So let's see what Gemini responds. So Gemini basically says that a large flock of birds suddenly takes flight from the ground in front of the vehicle, and the expected behavior is basically to slow down and, uh, remain alert and possibly adjust the

[5:20](https://www.youtube.com/watch?v=iS9YFW28XyM&t=320s) speed, uh, afterwards. So this is a very, basically, I would say, perfect answer to how we should drive with respect to this scenario. And this is another scenario where you can see there is a scooter rider in front of us, and unfortunately, uh, she slipped, right?

[5:38](https://www.youtube.com/watch?v=iS9YFW28XyM&t=338s) And this is because Ge-- basically, it's at night and there, there was rain earlier, so the, the road was wet. So these kind of events also happens, um, quite often, like when the scale is large, right?

[5:54](https://www.youtube.com/watch?v=iS9YFW28XyM&t=354s) But this is a very much a safety critical events that we, we really want to get it right and get it perfect. And let's ask Gemini again. And Gemini actually identified the entire scenario correctly, and even identify a, a gas station, uh, in faraway scene.

[6:11](https://www.youtube.com/watch?v=iS9YFW28XyM&t=371s) So I, I think it's very telling that in today's foundation models, they actually generalize well in these kind of rare events. And the questions becomes like, how, how do we, like how do we leverage this technology for autonomous driving?

[6:30](https://www.youtube.com/watch?v=iS9YFW28XyM&t=390s) So here is our exploration, uh, in research, where we want to have a more generalizable autonomous driving system by leveraging Gemini or other multimodal large language models.

[6:43](https://www.youtube.com/watch?v=iS9YFW28XyM&t=403s) So here we, we call this EMMA. So this is the simplest form of EMMA, where the idea is very straightforward. Yeah. Gemini is very good at generalizing in various scenarios.

[6:55](https://www.youtube.com/watch?v=iS9YFW28XyM&t=415s) So let's just put it to drive. So how do we do that? Um, so on the top left is about router, where we want to know where we are going in this current, uh, driving.

[7:10](https://www.youtube.com/watch?v=iS9YFW28XyM&t=430s) So for example, like you can think about it as a Google Map signal, and then it tell us, "You should turn left or turn right at the next intersection," something like that.

[7:19](https://www.youtube.com/watch?v=iS9YFW28XyM&t=439s) And then we translate it into text, just pure text. And then on the bottom left, we have the, the, the vehicle with surrounding cameras. There are eight cameras in this image, and it covers 360 degrees.

[7:33](https://www.youtube.com/watch?v=iS9YFW28XyM&t=453s) And then we just input this video and the text, the, the routing text to, to EMMA, basically built on top of Gemini, and then we ask Gemini how to drive in the next scenario, right?

[7:46](https://www.youtube.com/watch?v=iS9YFW28XyM&t=466s) So Gemini will output, um, future waypoints, which means the future locations of this car should be in in the next few seconds. So this, this, this method is very simple, right?

[7:58](https://www.youtube.com/watch?v=iS9YFW28XyM&t=478s) Like, and there are three major traits here. The first thing is that it is self-supervised, because whenever we have a driving log, right, we know where the car were.

[8:10](https://www.youtube.com/watch?v=iS9YFW28XyM&t=490s) So at any time point, we can use this formulation to train the model, because we know where the future the car should be in. So this is sup-- self-supervised, very scalable.

[8:21](https://www.youtube.com/watch?v=iS9YFW28XyM&t=501s) The second thing is it is camera only. We, we don't have lidar yet because, you know, Gemini is a camera model. So we only need cameras to drive. The third thing is that it is high-definition map free.

[8:33](https://www.youtube.com/watch?v=iS9YFW28XyM&t=513s) I think for people who are-- who understand autonomous driving technology to a certain extent, like we usually hear that Waymo's car needs a lot of map priors. So in this model, we actually don't need any maps besides Google Map.

[8:49](https://www.youtube.com/watch?v=iS9YFW28XyM&t=529s) So how, how was the performance here? So we, we basically conducted a-- on a research benchmark called nuScenes. This is one of the most popular open-loop planner benchmarks. And for this simple model, it achieves state-of-the-art quality compared to all other models at that time.

[9:06](https://www.youtube.com/watch?v=iS9YFW28XyM&t=546s) So basically, whatever like customized model, small model, large models, like with this simple formulation, we can already achieve the, the best quality.

[9:17](https://www.youtube.com/watch?v=iS9YFW28XyM&t=557s) So now we are thinking more towards like, okay, so the self-supervised method does work to some extent, but there are a lot of drawbacks that we want to basically remedy, right?

[9:28](https://www.youtube.com/watch?v=iS9YFW28XyM&t=568s) The first thing is that, okay, we have had a lot of labeled data, a lot of expert models. How do we leverage them to further improve the quality? The second thing is that there is a very clear drawback about end-to-end models where the explainability is not there, right?

[9:47](https://www.youtube.com/watch?v=iS9YFW28XyM&t=587s) Like we can only see the, the output of planner, but we don't know what happens inside. So here is what we do, right? We basically have a chain-of-thought process before outputting the, the planner.

[10:01](https://www.youtube.com/watch?v=iS9YFW28XyM&t=601s) So basically, we let the model explain itself about how it should drive in this scenario. For example, we ask the model to identify critical objects on the road first.

[10:13](https://www.youtube.com/watch?v=iS9YFW28XyM&t=613s) In this scenario, they, they identify the cyclist and the vehicle. And then we ask them to explain what kind of behaviors those critical objects will, will do and what kind of driving meta decision that we should, we should go for.

[10:27](https://www.youtube.com/watch?v=iS9YFW28XyM&t=627s) And in this scenario, it says we should just keep the normal speed. In some of the scenario, it will say we should yield or slow down, something like that.

[10:37](https://www.youtube.com/watch?v=iS9YFW28XyM&t=637s) And with this, we actually achieved an even better planner. Um, it's measured in our own Waymo Open Motion Dataset. And this, this dataset is actually hundred K dataset, about hundred times larger than the nuScenes dataset.

[10:52](https://www.youtube.com/watch?v=iS9YFW28XyM&t=652s) And more importantly is that the baselines are much stronger because they are not-- they, they are specialized models. They are Wayformer and MotionLM. So they are basically built on top of very sophisticated oracle perception system, and they takes inputs from, from those oracle perception and road graph, that's basically high-definition map, and traffic light states.

[11:16](https://www.youtube.com/watch?v=iS9YFW28XyM&t=676s) So they have inputs from everything almost, and then their own job is to output the planner. So this is-- these are very strong baselines. And with chain-of-thought reasoning, you can see that it actually comes on top.

[11:29](https://www.youtube.com/watch?v=iS9YFW28XyM&t=689s) So with this architecture, we actually, uh, see that. It actually performs very well in the academic setting.

[11:39](https://www.youtube.com/watch?v=iS9YFW28XyM&t=699s) So one promise in foundation model is that, you know, OpenAI created this kind of popularized the scaling laws. It basically says that the quality will keep improving if you have a larger model and larger dataset.

[11:52](https://www.youtube.com/watch?v=iS9YFW28XyM&t=712s) And we, we, we are showing here that if we train, like this is a dataset that's magnitude larger than any of the academic datasets that's released publicly. And we see that if we keep training, uh, more data- And we see the quality keeps improving.

[12:11](https://www.youtube.com/watch?v=iS9YFW28XyM&t=731s) So the, the y-axis is the perplexity here. So it basically means that the lower the perplexity, the better the planner results. So we see the quality can, can be further improved.

[12:24](https://www.youtube.com/watch?v=iS9YFW28XyM&t=744s) So, um, we talk about the explainability here, and one thing that we also are thinking about is that, uh, why not just train, um, various tasks, right? Like, because it is a vision language models at the core, and the language part is super flexible, we can basically formulate any types of tasks with it.

[12:48](https://www.youtube.com/watch?v=iS9YFW28XyM&t=768s) So, so we think about, like, let's make EMMA the most generalizable model possible, right? So we add a lot of different tasks into it. Uh, in this example, we, we demonstrate 3D detection, road graph estimation, and some kind of freeform VQA here.

[13:05](https://www.youtube.com/watch?v=iS9YFW28XyM&t=785s) And for any prompt on the left, uh, if you enter it, then EMMA will spit out the answers correspondingly. So it will have the first one is the, the end-to-end driving, the second one is 3D detection, et cetera.

[13:18](https://www.youtube.com/watch?v=iS9YFW28XyM&t=798s) And then we decode and visualize the results on the right.

[13:24](https://www.youtube.com/watch?v=iS9YFW28XyM&t=804s) So we also measure the, the detection quality here on our Waymo Open Dataset. So it actually achieves a very similar quality compared to other state-of-the-art models. So this demonstrates the EMMA's capability on generalizing two different tasks and co-training them together.

[13:43](https://www.youtube.com/watch?v=iS9YFW28XyM&t=823s) So here, here are some visualization of the EMMA's outputs, including the driving trajectory detection and road graph. So I think the, the predictions are, are reasonably well.

[13:57](https://www.youtube.com/watch?v=iS9YFW28XyM&t=837s) So now it seems like we have, um, some kind of prototype for, for end-to-end multimodal large language models. But then the, the question comes next is about how we evaluate it, validate the version, and make sure that it's safe.

[14:12](https://www.youtube.com/watch?v=iS9YFW28XyM&t=852s) So evaluation is part of, you know, the, the entire solution, right? We cannot really make, make this model succeed without any evaluation. And does... For evaluation previously, all the results I showed was about open loop evaluation, which means that we just replay the video and then see the...

[14:33](https://www.youtube.com/watch?v=iS9YFW28XyM&t=873s) check the model quality. But this is usually not the most faithful way to, to, to do evaluation, so there are simulations and real world road testing. And simulation means that we create a, a virtual world where we can test our model to drive the car in that world.

[14:52](https://www.youtube.com/watch?v=iS9YFW28XyM&t=892s) And real road testing is just deploying it. So the difficulties are different, and usually we trust simulation a lot more compared to open loop.

[15:04](https://www.youtube.com/watch?v=iS9YFW28XyM&t=904s) So one thing that we are also thinking about is that, yes, foundation models in the generative world are, are, is also very advanced. This is a Veo 2 from Google DeepMind, where we see the videos generated are very realistic.

[15:18](https://www.youtube.com/watch?v=iS9YFW28XyM&t=918s) So we are also thinking about maybe we can leverage it for simulation. So this is also, like, our latest research, um, sensor simulation for end-to-end driving evaluation. So for this model, we actually use a open sourced, um, generative models from Google, and then we generate the, the, the videos, and then we use that to basically place our

[15:42](https://www.youtube.com/watch?v=iS9YFW28XyM&t=942s) EMMA, and then we can evaluate the, the quality. And we can also control in various conditions, like we can change the weather, we can change the time, we can change a lot of different ways.

[15:54](https://www.youtube.com/watch?v=iS9YFW28XyM&t=954s) Uh, due to time, I basically removed those videos, so sorry about that, the visual part. So this is the results where we can test our EMMA planner in various conditions, right?

[16:06](https://www.youtube.com/watch?v=iS9YFW28XyM&t=966s) We can switch from rain to no rain. We can switch different time in day. And this is very much like,

[16:16](https://www.youtube.com/watch?v=iS9YFW28XyM&t=976s) um, aligns with our intuition, where when it rains, when the weather is bad, then usually our planner gets a little bit worse because it's, it's a camera-only model, so it, it affects the camera domain.

[16:29](https://www.youtube.com/watch?v=iS9YFW28XyM&t=989s) And then for time of day is also kind of aligns where, uh, at night usually the quality is worse compared in the, in the daytime. So usually at noon or at, in the afternoon, the model will perform the best.

[16:44](https://www.youtube.com/watch?v=iS9YFW28XyM&t=1004s) So this is the, the last slide. So these are all generated videos that we, we, we test our models in. So I think this is a ongoing research where I, I think it's a very exciting field where we have all the foundation models, and because Waymo and Google are both belongs to Alphabet, so we get to ac-

[17:04](https://www.youtube.com/watch?v=iS9YFW28XyM&t=1024s) get access to all these models for free, [chuckles] sort of. Yeah, so, so we, we basically try to adapt them and try to see if we can improve the generalization and, and help Waymo to scale to the next big thing.

[17:18](https://www.youtube.com/watch?v=iS9YFW28XyM&t=1038s) So thank you. Thank you for your attention. [audience applauding] [upbeat music]
