Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

Talk transcript

1. 0:00

   [on-hold music] So I, I think a lot of people here already took Waymo here in San Francisco.

2. 0:21

   So we drive every day, and I think the technology just works as you all witness. So, uh, I'm gonna talk a little bit about the history of autonomous driving research, like what it took to, to, to take us here and in the current status.

3. 0:36

   So in the autonomous driving history, uh, people started the research in 1980s, where it's just very simple neural networks. There are three layers. And over time, people start to think about, you know, going deeper and deeper.

4. 0:51

   And then in around, uh, 2020, there is papers from NVIDIA and other, uh, research labs publishing end-to-end driving models. And this is one of the videos that they published at that time, right?

5. 1:04

   And then we can, we can check it out, right? The, the, the car in front driving is the autonomous driving cars from the research labs at that time. And you can see that it's actually drifting, right?

6. 1:15

   So this is more like a L2, uh, sort of technology, and [chuckles] you, you wouldn't want to basically ride in it. So why is an L4 system that Waymo has so much better, right?

7. 1:27

   You can see it drives around the San Francisco in various regions, in downtown areas, and it avoids pedestrians, cyclists, and it makes roads very safe. So, so the secret is very basically on the top, where, uh, you can see that this is a sort of visualization or departing screen, where it visualize everything that the system understands.

8. 1:50

   So it can actually capture all the cars, pedestrians, cyclists, traffic lights, crossroads, and it understand almost everything related to driving. And we can see, uh, this is a very complicated system.

9. 2:05

   If we break it down, it's basically a perception system where it understands the world, and there is a prediction system that basically predicts the future of... future world states given the current one.

10. 2:18

    And finally, there is a planning system that basically tell us, like, how we should drive in this scenario. Like, we should turn right, how much acceleration, how much steering, et cetera.

11. 2:29

    So this is the, the whole autonomous driving system that is very complicated and dedicated. But it, it solves problems, right? It drives in San Francisco today. So what else?

12. 2:40

    So what are we heading with autonomous driving? The, the answer is scaling, right? So we've proven that we are operating in Phoenix, San Francisco, Austin, and Los Angeles today, and it's offering the rider-only service.

13. 2:57

    However, we are-- our ambition is not going to stop in just four cities now. So in this year, we are doing a road trip where we are visiting about 10 cities, um, in various locations.

14. 3:09

    Here is the map. And you can even see that we are actually going across the globe to, to Tokyo, Japan. So there is a lot of challenges when we start to scale.

15. 3:24

    So just recently, we finished a open challenge for participants to solve some of the hardest problems that we see in our spending. So let's replay the video a little bit here.

16. 3:38

    So on the, on the top left, there is a marathon run on the right, and then there is, uh, cones around the, the, the, the roads. And most interestingly is that when we drive, uh, closer to the intersection, if you look at the traffic light, it's actually red.

17. 3:55

    But there is a traffic control man waving us to go ahead. So it's actually green. [laughing] And, and this is very confusing, right? And all these long tails are very challenging to solve.

18. 4:06

    And there are challenges in, like, a lot of times that you've never seen it in your entire life of driving. But for Waymo, at Waymo scale, we will see very, very often, and we need to solve them to, in order to scale.

19. 4:23

    So one of the, the, the, the solutions arises where foundation model is, is very generalizable, right? So let's just take a example here that we feed in Gemini. Right, you can see the video where when we were driving peacefully, suddenly there are a, a, a bunch of angry birds, right?

20. 4:41

    They, they appear and attack our car virtually, right? So that's where we need to react. But for a car to understand the scenario, it also unders-- has to understand, okay, this is a fleet of birds and how they are going to behave, how they are going to interact with the car, right?

21. 4:58

    This is not the everyday driving that you see, but this is a critical long tail. So let's see what Gemini responds. So Gemini basically says that a large flock of birds suddenly takes flight from the ground in front of the vehicle, and the expected behavior is basically to slow down and, uh, remain alert and possibly adjust the

22. 5:20

    speed, uh, afterwards. So this is a very, basically, I would say, perfect answer to how we should drive with respect to this scenario. And this is another scenario where you can see there is a scooter rider in front of us, and unfortunately, uh, she slipped, right?

23. 5:38

    And this is because Ge-- basically, it's at night and there, there was rain earlier, so the, the road was wet. So these kind of events also happens, um, quite often, like when the scale is large, right?

24. 5:54

    But this is a very much a safety critical events that we, we really want to get it right and get it perfect. And let's ask Gemini again. And Gemini actually identified the entire scenario correctly, and even identify a, a gas station, uh, in faraway scene.

25. 6:11

    So I, I think it's very telling that in today's foundation models, they actually generalize well in these kind of rare events. And the questions becomes like, how, how do we, like how do we leverage this technology for autonomous driving?

26. 6:30

    So here is our exploration, uh, in research, where we want to have a more generalizable autonomous driving system by leveraging Gemini or other multimodal large language models.

27. 6:43

    So here we, we call this EMMA. So this is the simplest form of EMMA, where the idea is very straightforward. Yeah. Gemini is very good at generalizing in various scenarios.

28. 6:55

    So let's just put it to drive. So how do we do that? Um, so on the top left is about router, where we want to know where we are going in this current, uh, driving.

29. 7:10

    So for example, like you can think about it as a Google Map signal, and then it tell us, "You should turn left or turn right at the next intersection," something like that.

30. 7:19

    And then we translate it into text, just pure text. And then on the bottom left, we have the, the, the vehicle with surrounding cameras. There are eight cameras in this image, and it covers 360 degrees.

31. 7:33

    And then we just input this video and the text, the, the routing text to, to EMMA, basically built on top of Gemini, and then we ask Gemini how to drive in the next scenario, right?

32. 7:46

    So Gemini will output, um, future waypoints, which means the future locations of this car should be in in the next few seconds. So this, this, this method is very simple, right?

33. 7:58

    Like, and there are three major traits here. The first thing is that it is self-supervised, because whenever we have a driving log, right, we know where the car were.

34. 8:10

    So at any time point, we can use this formulation to train the model, because we know where the future the car should be in. So this is sup-- self-supervised, very scalable.

35. 8:21

    The second thing is it is camera only. We, we don't have lidar yet because, you know, Gemini is a camera model. So we only need cameras to drive. The third thing is that it is high-definition map free.

36. 8:33

    I think for people who are-- who understand autonomous driving technology to a certain extent, like we usually hear that Waymo's car needs a lot of map priors. So in this model, we actually don't need any maps besides Google Map.

37. 8:49

    So how, how was the performance here? So we, we basically conducted a-- on a research benchmark called nuScenes. This is one of the most popular open-loop planner benchmarks. And for this simple model, it achieves state-of-the-art quality compared to all other models at that time.

38. 9:06

    So basically, whatever like customized model, small model, large models, like with this simple formulation, we can already achieve the, the best quality.

39. 9:17

    So now we are thinking more towards like, okay, so the self-supervised method does work to some extent, but there are a lot of drawbacks that we want to basically remedy, right?

40. 9:28

    The first thing is that, okay, we have had a lot of labeled data, a lot of expert models. How do we leverage them to further improve the quality? The second thing is that there is a very clear drawback about end-to-end models where the explainability is not there, right?

41. 9:47

    Like we can only see the, the output of planner, but we don't know what happens inside. So here is what we do, right? We basically have a channel, so process before outputting the, the planner.

42. 10:01

    So basically, we let the model explain itself about how it should drive in this scenario. For example, we ask the model to identify critical objects on the road first.

43. 10:13

    In this scenario, they, they identify the cyclist and the vehicle. And then we ask them to explain what kind of behaviors those critical objects will, will do and what kind of driving meta decision that we should, we should go for.

44. 10:27

    And in this scenario, it says we should just keep the normal speed. In some of the scenario, it will say we should yield or slow down, something like that.

45. 10:37

    And with this, we actually achieved an even better planner. Um, it's measured in our own Waymo Open Motion Dataset. And this, this dataset is actually hundred K dataset, about hundred times larger than the nuScenes dataset.

46. 10:52

    And more importantly is that the baselines are much stronger because they are not-- they, they are specialized models. They are Wayformer and MotionLM. So they are basically built on top of very sophisticated oracle perception system, and they takes inputs from, from those oracle perception and raw graph, that's basically high-definition map, and traffic light states.

47. 11:16

    So they have inputs from everything almost, and then their own job is to output the planner. So this is-- these are very strong baselines. And with channel so reasoning, you can see that it actually comes on top.

48. 11:29

    So with this architecture, we actually, uh, see that. It actually performs very well in the academic setting.

49. 11:39

    So one promise in foundation model is that, you know, OpenAI created this kind of popularized the scaling laws. It basically says that the quality will keep improving if you have a larger model and larger dataset.

50. 11:52

    And we, we, we are showing here that if we train, like this is a dataset that's magnitude larger than any of the academic datasets that's released publicly. And we see that if we keep training, uh, more data- And we see the quality keeps improving.

51. 12:11

    So the, the y-axis is the perplexity here. So it basically means that the lower the perplexity, the better the planner results. So we see the quality can, can be further improved.

52. 12:24

    So, um, we talk about the expandability here, and one thing that we also are thinking about is that, uh, why not just train, um, various tasks, right? Like, because it is a vision language models at the core, and the language part is super flexible, we can basically formulate any types of tasks with it.

53. 12:48

    So, so we think about, like, let's make EMMA the most generalizable model possible, right? So we add a lot of different tasks into it. Uh, in this example, we, we demonstrate 3D detection, road graph estimation, and some kind of freeform VQA here.

54. 13:05

    And for any prompt on the left, uh, if you enter it, then EMMA will spit out the answers correspondingly. So it will have the first one is the, the end-to-end driving, the second one is 3D detection, et cetera.

55. 13:18

    And then we decode and visualize the results on the right.

56. 13:24

    So we also measure the, the detection quality here on our Waymo Open Dataset. So it actually achieves a very similar quality compared to other state-of-the-art models. So this demonstrates the EMMA's capability on generalizing two different tasks and co-training them together.

57. 13:43

    So here, here are some visualization of the EMMA's outputs, including the driving trajectory detection and road graph. So I think the, the predictions are, are reasonably well.

58. 13:57

    So now it seems like we have, um, some kind of prototype for, for end-to-end multimodal large language models. But then the, the question comes next is about how we evaluate it, validate the version, and make sure that it's safe.

59. 14:12

    So evaluation is part of, you know, the, the entire solution, right? We cannot really make, make this model succeed without any evaluation. And does... For evaluation previously, all the results I showed was about open loop evaluation, which means that we just replay the video and then see the...

60. 14:33

    check the model quality. But this is usually not the most feasible way to, to, to do evaluation, so there are simulations and real world road testing. And simulation means that we create a, a virtual world where we can test our model to drive the car in that world.

61. 14:52

    And real road testing is just deploying it. So the difficulties are different, and usually we trust simulation a lot more compared to open world.

62. 15:04

    So one thing that we are also thinking about is that, yes, foundation models in the generative world are, are, is also very advanced. This is a Veo 2 from Google DeepMind, where we see the videos generated are very realistic.

63. 15:18

    So we are also thinking about maybe we can leverage it for simulation. So this is also, like, our latest research, um, sensor simulation for end-to-end driving evaluation. So for this model, we actually use a open sourced, um, generative models from Google, and then we generate the, the, the videos, and then we use that to basically place our

64. 15:42

    EMMA, and then we can evaluate the, the quality. And we can also control in various conditions, like we can change the weather, we can change the time, we can change a lot of different ways.

65. 15:54

    Uh, due to time, I basically removed those videos, so sorry about that, the visual part. So this is the results where we can test our EMMA planner in various conditions, right?

66. 16:06

    We can switch from rain to no rain. We can switch different time in day. And this is very much like,

67. 16:16

    um, aligns with our intuition, where when it rains, when the weather is bad, then usually our planner gets a little bit worse because it's, it's a camera-only model, so it, it affects the camera domain.

68. 16:29

    And then for time of day is also kind of aligns where, uh, at night usually the quality is worse compared in the, in the daytime. So usually at noon or at, in the afternoon, the model will perform the best.

69. 16:44

    So this is the, the last slide. So these are all generated videos that we, we, we test our models in. So I think this is a ongoing research where I, I think it's a very exciting field where we have all the foundation models, and because Waymo and Google are both belongs to Alphabet, so we get to ac-

70. 17:04

    get access to all these models for free, [chuckles] sort of. Yeah, so, so we, we basically try to adapt them and try to see if we can improve the generalization and, and help Waymo to scale to the next big thing.

71. 17:18

    So thank you. Thank you for your attention. [audience applauding] [upbeat music]