Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

A research car drifts across its path in the opening driving demonstration. What separates that behavior from a vehicle that can navigate San Francisco without a driver? Jyh-Jing Hwang approaches the question through a brief history: three-layer neural networks in the 1980s, followed by increasingly deep models and end-to-end driving research. He places the illustrated wave of research around 2020, although NVIDIA’s camera-to-steering work dates to 2016. Hwang characterizes the drifting demonstration as L2-like, then contrasts it with Waymo’s L4 system.

The San Francisco demonstration shows more than street footage. Above it, a debugging view displays the system’s understanding of the scene: cars, pedestrians, cyclists, traffic lights and intersections. This explicit representation helps explain the complexity behind apparently ordinary driving.

The dedicated driving stack divides the problem into three responsibilities:

Perception supplies the scene that prediction projects forward; planning then decides how the vehicle should move through it. This is a complicated system, but it already solves the practical problem of driving in San Francisco.

The next problem is scaling. At the time of the talk, Hwang describes rider-only service in Phoenix, San Francisco, Austin and Los Angeles, alongside a road trip visiting about ten cities that year. The expansion map reaches Tokyo, too. That destination represents exploration and development, not another rider-only service location.

A recently completed open challenge illustrates the difficulty. A marathon runner appears beside a road lined with cones. As the vehicle approaches an intersection, the traffic light is red—but a traffic controller waves it onward. The light has not changed; the relevant instruction comes from the person directing traffic. Understanding the scene requires interpreting these signals together, rather than treating the red light as an isolated observation.

A situation that an individual driver never encounters can still recur across a large fleet. Geographic expansion therefore demands more than recognizing familiar road objects. The system must handle unusual combinations of objects, instructions and behavior.

Foundation models offer a possible source of that broader understanding. In the first demonstration, Gemini receives a driving video in which a flock of birds suddenly takes flight in front of the vehicle. Recognizing birds is only part of the task: a useful driving interpretation must also anticipate how they could interact with the car. Gemini describes the event and recommends slowing down, remaining alert and adjusting speed as needed. Hwang calls this a “perfect answer” for the scenario. The displayed slide identifies the model as Gemini 2.5 Pro.

The next video shows a scooter rider slipping at night on a road wet from earlier rain. Hwang reports that Gemini identifies the incident correctly and even notices a distant gas station. These are qualitative demonstrations of scene understanding, but they motivate a concrete research question: how can that understanding become an input to driving, rather than remain a description of what happened?

EMMA, the End-to-End Multimodal Model for Autonomous Driving, begins with a simple research formulation built on Gemini. A router supplies high-level navigation guidance, such as turning left or right at the next intersection. That guidance becomes ordinary text. The illustrated vehicle has eight cameras covering 360 degrees; their video and the routing text enter the model together. EMMA outputs future waypoints: positions the vehicle should occupy over the next few seconds.

Driving logs provide the training targets. At a chosen point in a recorded drive, the model receives the available observations and routing instruction. The vehicle’s subsequent recorded positions supply the desired output. That makes the basic trajectory-learning task self-supervised: it does not require someone to label a new ideal trajectory for every training example.

The essential target construction can be expressed by splitting a position log at the current frame. Here, positions contains illustrative local coordinates, and the three later positions become the future-waypoint target:

positions = [
    (0.0, 0.0),
    (1.0, 0.0),
    (2.0, 0.1),
    (3.0, 0.3),
    (4.0, 0.6),
]
current_frame = 1
horizon = 3

future_waypoints = positions[
    current_frame + 1 : current_frame + 1 + horizon
]
assert len(future_waypoints) == horizon

The important boundary is temporal: future positions are training targets, while the model’s inputs describe what is available when it must make the decision.

Two other properties distinguish this prototype:

• Camera-only sensing: this formulation does not use lidar; its sensory input matches the visual capabilities of the underlying model.
• No high-definition map: EMMA does not require the detailed map priors associated with a dedicated driving stack. It still receives ordinary routing guidance, which Hwang likens to Google Maps.

These are properties of the research model, not a description of the sensing configuration of Waymo’s deployed service.

Hwang reports state-of-the-art planning quality for the simple formulation on the nuScenes open-loop benchmark against the contemporary models in his comparison. The comparison is version-sensitive: the original EMMA paper distinguishes the base model from an internally pretrained EMMA+, so the reported headline should not be read as an unconditional win for every EMMA configuration.

The simple formulation leaves two opportunities unresolved. Waymo already has labeled data and expert models that could improve training, and a trajectory alone reveals little about why an end-to-end model chose it. EMMA’s next extension produces a driving rationale before the planned waypoints.

The reasoning sequence makes intermediate judgments visible:

1. Identify the critical road objects.
2. Describe their expected behavior.
3. Choose a high-level driving decision.
4. Output the future trajectory.

In the illustrated scene, the model identifies a cyclist and a vehicle, considers their behavior, and chooses to maintain normal speed. Other scenes can call for yielding or slowing down. The rationale provides something to inspect alongside the trajectory, although its presence alone does not establish that it faithfully explains every internal computation.

The next evaluation uses the Waymo Open Motion Dataset. Hwang describes its scale as roughly 100,000 scenarios, about 100 times the scene count of nuScenes. The specialized baselines, Wayformer and MotionLM, receive privileged scene information: oracle perception, a road graph serving as a high-definition map, and traffic-light states. Their task is to produce the driving output from these already structured inputs.

Hwang reports that EMMA with reasoning comes out ahead of these specialized baselines in the academic comparison he presents. That claim belongs to the presented experiment: the original paper separates its WOMD comparisons from its internal-data reasoning experiments, and its WOMD results depend on pretraining and prediction horizon. The useful architectural comparison is that a model working from visual and language inputs can compete with systems supplied with much richer structured scene information.

Hwang next connects EMMA to scaling laws, crediting OpenAI with popularizing the relationship between model size, dataset size and quality. His experiment trains on a dataset substantially larger than the public academic datasets discussed earlier. Hwang reports that increasing training data continues to lower planner perplexity in the displayed experiment.

The vertical axis matters: it is perplexity, which Hwang interprets as a measure of planner quality, with lower values better. This supports a training-scaling trend; it does not supply a numerical conversion into collision rates or safety in interactive driving. Those questions require a different evaluation setup.

A language interface also makes the task itself flexible. Instead of training EMMA only to produce a trajectory, the researchers add tasks selected through prompts:

The prompt determines what EMMA should return. Its outputs are then decoded into visualizations, including trajectories, detections and road graphs. The same underlying model can therefore learn both how to move through a scene and how to describe its structure.

Hwang reports detection quality similar to other state-of-the-art models on the Waymo Open Dataset. The original paper’s detection comparison uses camera-primary LET matching with precision, recall and F1, rather than standard average precision. Hwang presents the result as evidence that different driving tasks can be co-trained, then shows their outputs together: a planned path, detected objects and an estimated road graph.

At this point, EMMA is a research prototype with encouraging results. The next question is how to evaluate a version and establish whether its driving is safe. Hwang identifies all the preceding results as open-loop evaluation: replay recorded video and assess the model’s output. The vehicle’s proposed actions do not determine what the recorded camera sees next.

The evaluation methods differ in how they expose the consequences of driving decisions:

Hwang places greater trust in simulation than in open-loop replay, while recognizing that each method has different difficulties. A simulator can test what follows from the model’s actions; real-road testing adds the demands and risks of physical deployment.

Generative video models suggest another route into evaluation. Hwang first shows Veo 2 from Google DeepMind as an example of realistic video generation, then introduces research on sensor simulation for end-to-end driving. He describes that work as using an open-sourced Google generative model to create videos for EMMA evaluation; he does not identify it as Veo 2. The distinction matters because the example of video realism and the actual simulation implementation are separate parts of the presentation.

The generated inputs can vary conditions such as weather and time of day. This gives the evaluator a way to ask how the planner behaves when the visual environment changes. Hwang notes that some demonstration videos were removed for time, then presents results for switching between rain and no rain and across times of day.

Hwang reports worse planner quality in rain or bad weather, attributing the degradation to changes in the camera input domain. He also reports worse quality at night than during the day, with the best performance around noon or afternoon. For a camera-only model, these experiments probe a direct dependency: the planner must derive its decisions from imagery whose visibility and appearance change with the conditions.

The final montage shows generated road videos used to test the models, spanning different streets and lighting conditions. Hwang closes with this work still in progress. Shared Alphabet ownership gives Waymo access to Google’s foundation models, which the researchers are adapting both to interpret driving scenes and to create environments in which planners can be evaluated. The intended result is better generalization—the capability needed to extend autonomous driving beyond the places and situations already mastered.