AI Engineer World's Fair 2025
Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo
Read the talk
Waymo’s EMMA: From Camera Video to Driving Decisions
EMMA turns camera video and routing text into future waypoints, then extends that formulation with driving rationales, multiple tasks and generated environments for evaluation.
From a talk by Jyh-Jing Hwang
Before you start: Basic familiarity with machine-learning inputs, training targets and model evaluation is helpful; no autonomous-driving background is required.
Why does one car drift while another navigates San Francisco?
A research car drifts across its path in the opening driving demonstration. What separates that behavior from a vehicle that can navigate San Francisco without a driver? Jyh-Jing Hwang approaches the question through a brief history: three-layer neural networks in the 1980s, followed by increasingly deep models and end-to-end driving research. He places the illustrated wave of research around 2020, although NVIDIA’s camera-to-steering work dates to 2016. Hwang characterizes the drifting demonstration as L2-like, then contrasts it with Waymo’s L4 system.
The San Francisco demonstration shows more than street footage. Above it, a debugging view displays the system’s understanding of the scene: cars, pedestrians, cyclists, traffic lights and intersections. This explicit representation helps explain the complexity behind apparently ordinary driving.
The dedicated driving stack divides the problem into three responsibilities:
| Component | Responsibility |
|---|---|
| Perception | Understand the current world |
| Prediction | Forecast future world states |
| Planning | Choose turns, acceleration and steering |
Perception supplies the scene that prediction projects forward; planning then decides how the vehicle should move through it. This is a complicated system, but it already solves the practical problem of driving in San Francisco.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Geographic scale makes rare events unavoidable
The next problem is scaling. At the time of the talk, Hwang describes rider-only service in Phoenix, San Francisco, Austin and Los Angeles, alongside a road trip visiting about ten cities that year. The expansion map reaches Tokyo, too. That destination represents exploration and development, not another rider-only service location.
A recently completed open challenge illustrates the difficulty. A marathon runner appears beside a road lined with cones. As the vehicle approaches an intersection, the traffic light is red—but a traffic controller waves it onward. The light has not changed; the relevant instruction comes from the person directing traffic. Understanding the scene requires interpreting these signals together, rather than treating the red light as an isolated observation.
A situation that an individual driver never encounters can still recur across a large fleet. Geographic expansion therefore demands more than recognizing familiar road objects. The system must handle unusual combinations of objects, instructions and behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognizing birds and a fallen scooter rider
Foundation models offer a possible source of that broader understanding. In the first demonstration, Gemini receives a driving video in which a flock of birds suddenly takes flight in front of the vehicle. Recognizing birds is only part of the task: a useful driving interpretation must also anticipate how they could interact with the car. Gemini describes the event and recommends slowing down, remaining alert and adjusting speed as needed. Hwang calls this a “perfect answer” for the scenario. The displayed slide identifies the model as Gemini 2.5 Pro.
The next video shows a scooter rider slipping at night on a road wet from earlier rain. Hwang reports that Gemini identifies the incident correctly and even notices a distant gas station. These are qualitative demonstrations of scene understanding, but they motivate a concrete research question: how can that understanding become an input to driving, rather than remain a description of what happened?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turning video and routing text into waypoints
EMMA, the End-to-End Multimodal Model for Autonomous Driving, begins with a simple research formulation built on Gemini. A router supplies high-level navigation guidance, such as turning left or right at the next intersection. That guidance becomes ordinary text. The illustrated vehicle has eight cameras covering 360 degrees; their video and the routing text enter the model together. EMMA outputs future waypoints: positions the vehicle should occupy over the next few seconds.
Driving logs provide the training targets. At a chosen point in a recorded drive, the model receives the available observations and routing instruction. The vehicle’s subsequent recorded positions supply the desired output. That makes the basic trajectory-learning task self-supervised: it does not require someone to label a new ideal trajectory for every training example.
The essential target construction can be expressed by splitting a position log at the current frame. Here, positions contains illustrative local coordinates, and the three later positions become the future-waypoint target:
python
positions = [
(0.0, 0.0),
(1.0, 0.0),
(2.0, 0.1),
(3.0, 0.3),
(4.0, 0.6),
]
current_frame = 1
horizon = 3
future_waypoints = positions[
current_frame + 1 : current_frame + 1 + horizon
]
assert len(future_waypoints) == horizon
The important boundary is temporal: future positions are training targets, while the model’s inputs describe what is available when it must make the decision.
Two other properties distinguish this prototype:
- Camera-only sensing: this formulation does not use lidar; its sensory input matches the visual capabilities of the underlying model.
- No high-definition map: EMMA does not require the detailed map priors associated with a dedicated driving stack. It still receives ordinary routing guidance, which Hwang likens to Google Maps.
These are properties of the research model, not a description of the sensing configuration of Waymo’s deployed service.
Hwang reports state-of-the-art planning quality for the simple formulation on the nuScenes open-loop benchmark against the contemporary models in his comparison. The comparison is version-sensitive: the original EMMA paper distinguishes the base model from an internally pretrained EMMA+, so the reported headline should not be read as an unconditional win for every EMMA configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Producing a rationale before the trajectory
The simple formulation leaves two opportunities unresolved. Waymo already has labeled data and expert models that could improve training, and a trajectory alone reveals little about why an end-to-end model chose it. EMMA’s next extension produces a driving rationale before the planned waypoints.
The reasoning sequence makes intermediate judgments visible:
- Identify the critical road objects.
- Describe their expected behavior.
- Choose a high-level driving decision.
- Output the future trajectory.
In the illustrated scene, the model identifies a cyclist and a vehicle, considers their behavior, and chooses to maintain normal speed. Other scenes can call for yielding or slowing down. The rationale provides something to inspect alongside the trajectory, although its presence alone does not establish that it faithfully explains every internal computation.
The next evaluation uses the Waymo Open Motion Dataset. Hwang describes its scale as roughly 100,000 scenarios, about 100 times the scene count of nuScenes. The specialized baselines, Wayformer and MotionLM, receive privileged scene information: oracle perception, a road graph serving as a high-definition map, and traffic-light states. Their task is to produce the driving output from these already structured inputs.
Hwang reports that EMMA with reasoning comes out ahead of these specialized baselines in the academic comparison he presents. That claim belongs to the presented experiment: the original paper separates its WOMD comparisons from its internal-data reasoning experiments, and its WOMD results depend on pretraining and prediction horizon. The useful architectural comparison is that a model working from visual and language inputs can compete with systems supplied with much richer structured scene information.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More training data, lower planner perplexity
Hwang next connects EMMA to scaling laws, crediting OpenAI with popularizing the relationship between model size, dataset size and quality. His experiment trains on a dataset substantially larger than the public academic datasets discussed earlier. Hwang reports that increasing training data continues to lower planner perplexity in the displayed experiment.
The vertical axis matters: it is perplexity, which Hwang interprets as a measure of planner quality, with lower values better. This supports a training-scaling trend; it does not supply a numerical conversion into collision rates or safety in interactive driving. Those questions require a different evaluation setup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One model, several driving tasks
A language interface also makes the task itself flexible. Instead of training EMMA only to produce a trajectory, the researchers add tasks selected through prompts:
| Task | Requested output |
|---|---|
| End-to-end driving | Future vehicle trajectory |
| 3D detection | Objects in three-dimensional space |
| Road graph estimation | Road structure |
| Visual question answering | Freeform answers about the scene |
The prompt determines what EMMA should return. Its outputs are then decoded into visualizations, including trajectories, detections and road graphs. The same underlying model can therefore learn both how to move through a scene and how to describe its structure.
Hwang reports detection quality similar to other state-of-the-art models on the Waymo Open Dataset. The original paper’s detection comparison uses camera-primary LET matching with precision, recall and F1, rather than standard average precision. Hwang presents the result as evidence that different driving tasks can be co-trained, then shows their outputs together: a planned path, detected objects and an estimated road graph.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A recorded scene cannot test every consequence
At this point, EMMA is a research prototype with encouraging results. The next question is how to evaluate a version and establish whether its driving is safe. Hwang identifies all the preceding results as open-loop evaluation: replay recorded video and assess the model’s output. The vehicle’s proposed actions do not determine what the recorded camera sees next.
The evaluation methods differ in how they expose the consequences of driving decisions:
| Method | What happens |
|---|---|
| Open-loop replay | Assess outputs on a fixed recording |
| Simulation | Let the model drive in a virtual world |
| Real-road testing | Deploy the model in the physical world |
Hwang places greater trust in simulation than in open-loop replay, while recognizing that each method has different difficulties. A simulator can test what follows from the model’s actions; real-road testing adds the demands and risks of physical deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Changing rain and time of day
Generative video models suggest another route into evaluation. Hwang first shows Veo 2 from Google DeepMind as an example of realistic video generation, then introduces research on sensor simulation for end-to-end driving. He describes that work as using an open-sourced Google generative model to create videos for EMMA evaluation; he does not identify it as Veo 2. The distinction matters because the example of video realism and the actual simulation implementation are separate parts of the presentation.
The generated inputs can vary conditions such as weather and time of day. This gives the evaluator a way to ask how the planner behaves when the visual environment changes. Hwang notes that some demonstration videos were removed for time, then presents results for switching between rain and no rain and across times of day.
Hwang reports worse planner quality in rain or bad weather, attributing the degradation to changes in the camera input domain. He also reports worse quality at night than during the day, with the best performance around noon or afternoon. For a camera-only model, these experiments probe a direct dependency: the planner must derive its decisions from imagery whose visibility and appearance change with the conditions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adapting foundation models for broader driving
The final montage shows generated road videos used to test the models, spanning different streets and lighting conditions. Hwang closes with this work still in progress. Shared Alphabet ownership gives Waymo access to Google’s foundation models, which the researchers are adapting both to interpret driving scenes and to create environments in which planners can be evaluated. The intended result is better generalization—the capability needed to extend autonomous driving beyond the places and situations already mastered.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Waymo's introduction to EMMA, its shared language interface, multitask learning and deployment challenges.
Original formulation, benchmark tables and limitations.
Attention-based motion forecasting with alternative strategies for combining scene information.
Further reading
An early camera-to-steering system trained from human driving examples.
Updates since the talk
- Drive&GenPaper
Later publication describing controlled driving-video generation and tests of planner behavior on real versus generated inputs.
Read the complete timestamped transcript
- 0:00
[on-hold music] So I, I think a lot of people here already took Waymo here in San Francisco.
- 0:21
So we drive every day, and I think the technology just works as you all witness. So, uh, I'm gonna talk a little bit about the history of autonomous driving research, like what it took to, to, to take us here and in the current status.
- 0:36
So in the autonomous driving history, uh, people started the research in 1980s, where it's just very simple neural networks. There are three layers. And over time, people start to think about, you know, going deeper and deeper.
- 0:51
And then in around, uh, 2020, there is papers from NVIDIA and other, uh, research labs publishing end-to-end driving models. And this is one of the videos that they published at that time, right?
- 1:04
And then we can, we can check it out, right? The, the, the car in front driving is the autonomous driving cars from the research labs at that time. And you can see that it's actually drifting, right?
- 1:15
So this is more like a L2, uh, sort of technology, and [chuckles] you, you wouldn't want to basically ride in it. So why is an L4 system that Waymo has so much better, right?
- 1:27
You can see it drives around the San Francisco in various regions, in downtown areas, and it avoids pedestrians, cyclists, and it makes roads very safe. So, so the secret is very basically on the top, where, uh, you can see that this is a sort of visualization or departing screen, where it visualize everything that the system understands.
- 1:50
So it can actually capture all the cars, pedestrians, cyclists, traffic lights, crossroads, and it understand almost everything related to driving. And we can see, uh, this is a very complicated system.
- 2:05
If we break it down, it's basically a perception system where it understands the world, and there is a prediction system that basically predicts the future of... future world states given the current one.
- 2:18
And finally, there is a planning system that basically tell us, like, how we should drive in this scenario. Like, we should turn right, how much acceleration, how much steering, et cetera.
- 2:29
So this is the, the whole autonomous driving system that is very complicated and dedicated. But it, it solves problems, right? It drives in San Francisco today. So what else?
- 2:40
So what are we heading with autonomous driving? The, the answer is scaling, right? So we've proven that we are operating in Phoenix, San Francisco, Austin, and Los Angeles today, and it's offering the rider-only service.
- 2:57
However, we are-- our ambition is not going to stop in just four cities now. So in this year, we are doing a road trip where we are visiting about 10 cities, um, in various locations.
- 3:09
Here is the map. And you can even see that we are actually going across the globe to, to Tokyo, Japan. So there is a lot of challenges when we start to scale.
- 3:24
So just recently, we finished a open challenge for participants to solve some of the hardest problems that we see in our spending. So let's replay the video a little bit here.
- 3:38
So on the, on the top left, there is a marathon run on the right, and then there is, uh, cones around the, the, the, the roads. And most interestingly is that when we drive, uh, closer to the intersection, if you look at the traffic light, it's actually red.
- 3:55
But there is a traffic control man waving us to go ahead. So it's actually green. [laughing] And, and this is very confusing, right? And all these long tails are very challenging to solve.
- 4:06
And there are challenges in, like, a lot of times that you've never seen it in your entire life of driving. But for Waymo, at Waymo scale, we will see very, very often, and we need to solve them to, in order to scale.
- 4:23
So one of the, the, the, the solutions arises where foundation model is, is very generalizable, right? So let's just take a example here that we feed in Gemini. Right, you can see the video where when we were driving peacefully, suddenly there are a, a, a bunch of angry birds, right?
- 4:41
They, they appear and attack our car virtually, right? So that's where we need to react. But for a car to understand the scenario, it also unders-- has to understand, okay, this is a fleet of birds and how they are going to behave, how they are going to interact with the car, right?
- 4:58
This is not the everyday driving that you see, but this is a critical long tail. So let's see what Gemini responds. So Gemini basically says that a large flock of birds suddenly takes flight from the ground in front of the vehicle, and the expected behavior is basically to slow down and, uh, remain alert and possibly adjust the
- 5:20
speed, uh, afterwards. So this is a very, basically, I would say, perfect answer to how we should drive with respect to this scenario. And this is another scenario where you can see there is a scooter rider in front of us, and unfortunately, uh, she slipped, right?
- 5:38
And this is because Ge-- basically, it's at night and there, there was rain earlier, so the, the road was wet. So these kind of events also happens, um, quite often, like when the scale is large, right?
- 5:54
But this is a very much a safety critical events that we, we really want to get it right and get it perfect. And let's ask Gemini again. And Gemini actually identified the entire scenario correctly, and even identify a, a gas station, uh, in faraway scene.
- 6:11
So I, I think it's very telling that in today's foundation models, they actually generalize well in these kind of rare events. And the questions becomes like, how, how do we, like how do we leverage this technology for autonomous driving?
- 6:30
So here is our exploration, uh, in research, where we want to have a more generalizable autonomous driving system by leveraging Gemini or other multimodal large language models.
- 6:43
So here we, we call this EMMA. So this is the simplest form of EMMA, where the idea is very straightforward. Yeah. Gemini is very good at generalizing in various scenarios.
- 6:55
So let's just put it to drive. So how do we do that? Um, so on the top left is about router, where we want to know where we are going in this current, uh, driving.
- 7:10
So for example, like you can think about it as a Google Map signal, and then it tell us, "You should turn left or turn right at the next intersection," something like that.
- 7:19
And then we translate it into text, just pure text. And then on the bottom left, we have the, the, the vehicle with surrounding cameras. There are eight cameras in this image, and it covers 360 degrees.
- 7:33
And then we just input this video and the text, the, the routing text to, to EMMA, basically built on top of Gemini, and then we ask Gemini how to drive in the next scenario, right?
- 7:46
So Gemini will output, um, future waypoints, which means the future locations of this car should be in in the next few seconds. So this, this, this method is very simple, right?
- 7:58
Like, and there are three major traits here. The first thing is that it is self-supervised, because whenever we have a driving log, right, we know where the car were.
- 8:10
So at any time point, we can use this formulation to train the model, because we know where the future the car should be in. So this is sup-- self-supervised, very scalable.
- 8:21
The second thing is it is camera only. We, we don't have lidar yet because, you know, Gemini is a camera model. So we only need cameras to drive. The third thing is that it is high-definition map free.
- 8:33
I think for people who are-- who understand autonomous driving technology to a certain extent, like we usually hear that Waymo's car needs a lot of map priors. So in this model, we actually don't need any maps besides Google Map.
- 8:49
So how, how was the performance here? So we, we basically conducted a-- on a research benchmark called nuScenes. This is one of the most popular open-loop planner benchmarks. And for this simple model, it achieves state-of-the-art quality compared to all other models at that time.
- 9:06
So basically, whatever like customized model, small model, large models, like with this simple formulation, we can already achieve the, the best quality.
- 9:17
So now we are thinking more towards like, okay, so the self-supervised method does work to some extent, but there are a lot of drawbacks that we want to basically remedy, right?
- 9:28
The first thing is that, okay, we have had a lot of labeled data, a lot of expert models. How do we leverage them to further improve the quality? The second thing is that there is a very clear drawback about end-to-end models where the explainability is not there, right?
- 9:47
Like we can only see the, the output of planner, but we don't know what happens inside. So here is what we do, right? We basically have a channel, so process before outputting the, the planner.
- 10:01
So basically, we let the model explain itself about how it should drive in this scenario. For example, we ask the model to identify critical objects on the road first.
- 10:13
In this scenario, they, they identify the cyclist and the vehicle. And then we ask them to explain what kind of behaviors those critical objects will, will do and what kind of driving meta decision that we should, we should go for.
- 10:27
And in this scenario, it says we should just keep the normal speed. In some of the scenario, it will say we should yield or slow down, something like that.
- 10:37
And with this, we actually achieved an even better planner. Um, it's measured in our own Waymo Open Motion Dataset. And this, this dataset is actually hundred K dataset, about hundred times larger than the nuScenes dataset.
- 10:52
And more importantly is that the baselines are much stronger because they are not-- they, they are specialized models. They are Wayformer and MotionLM. So they are basically built on top of very sophisticated oracle perception system, and they takes inputs from, from those oracle perception and raw graph, that's basically high-definition map, and traffic light states.
- 11:16
So they have inputs from everything almost, and then their own job is to output the planner. So this is-- these are very strong baselines. And with channel so reasoning, you can see that it actually comes on top.
- 11:29
So with this architecture, we actually, uh, see that. It actually performs very well in the academic setting.
- 11:39
So one promise in foundation model is that, you know, OpenAI created this kind of popularized the scaling laws. It basically says that the quality will keep improving if you have a larger model and larger dataset.
- 11:52
And we, we, we are showing here that if we train, like this is a dataset that's magnitude larger than any of the academic datasets that's released publicly. And we see that if we keep training, uh, more data- And we see the quality keeps improving.
- 12:11
So the, the y-axis is the perplexity here. So it basically means that the lower the perplexity, the better the planner results. So we see the quality can, can be further improved.
- 12:24
So, um, we talk about the expandability here, and one thing that we also are thinking about is that, uh, why not just train, um, various tasks, right? Like, because it is a vision language models at the core, and the language part is super flexible, we can basically formulate any types of tasks with it.
- 12:48
So, so we think about, like, let's make EMMA the most generalizable model possible, right? So we add a lot of different tasks into it. Uh, in this example, we, we demonstrate 3D detection, road graph estimation, and some kind of freeform VQA here.
- 13:05
And for any prompt on the left, uh, if you enter it, then EMMA will spit out the answers correspondingly. So it will have the first one is the, the end-to-end driving, the second one is 3D detection, et cetera.
- 13:18
And then we decode and visualize the results on the right.
- 13:24
So we also measure the, the detection quality here on our Waymo Open Dataset. So it actually achieves a very similar quality compared to other state-of-the-art models. So this demonstrates the EMMA's capability on generalizing two different tasks and co-training them together.
- 13:43
So here, here are some visualization of the EMMA's outputs, including the driving trajectory detection and road graph. So I think the, the predictions are, are reasonably well.
- 13:57
So now it seems like we have, um, some kind of prototype for, for end-to-end multimodal large language models. But then the, the question comes next is about how we evaluate it, validate the version, and make sure that it's safe.
- 14:12
So evaluation is part of, you know, the, the entire solution, right? We cannot really make, make this model succeed without any evaluation. And does... For evaluation previously, all the results I showed was about open loop evaluation, which means that we just replay the video and then see the...
- 14:33
check the model quality. But this is usually not the most feasible way to, to, to do evaluation, so there are simulations and real world road testing. And simulation means that we create a, a virtual world where we can test our model to drive the car in that world.
- 14:52
And real road testing is just deploying it. So the difficulties are different, and usually we trust simulation a lot more compared to open world.
- 15:04
So one thing that we are also thinking about is that, yes, foundation models in the generative world are, are, is also very advanced. This is a Veo 2 from Google DeepMind, where we see the videos generated are very realistic.
- 15:18
So we are also thinking about maybe we can leverage it for simulation. So this is also, like, our latest research, um, sensor simulation for end-to-end driving evaluation. So for this model, we actually use a open sourced, um, generative models from Google, and then we generate the, the, the videos, and then we use that to basically place our
- 15:42
EMMA, and then we can evaluate the, the quality. And we can also control in various conditions, like we can change the weather, we can change the time, we can change a lot of different ways.
- 15:54
Uh, due to time, I basically removed those videos, so sorry about that, the visual part. So this is the results where we can test our EMMA planner in various conditions, right?
- 16:06
We can switch from rain to no rain. We can switch different time in day. And this is very much like,
- 16:16
um, aligns with our intuition, where when it rains, when the weather is bad, then usually our planner gets a little bit worse because it's, it's a camera-only model, so it, it affects the camera domain.
- 16:29
And then for time of day is also kind of aligns where, uh, at night usually the quality is worse compared in the, in the daytime. So usually at noon or at, in the afternoon, the model will perform the best.
- 16:44
So this is the, the last slide. So these are all generated videos that we, we, we test our models in. So I think this is a ongoing research where I, I think it's a very exciting field where we have all the foundation models, and because Waymo and Google are both belongs to Alphabet, so we get to ac-
- 17:04
get access to all these models for free, [chuckles] sort of. Yeah, so, so we, we basically try to adapt them and try to see if we can improve the generalization and, and help Waymo to scale to the next big thing.
- 17:18
So thank you. Thank you for your attention. [audience applauding] [upbeat music]