Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

Talk transcript

1. 0:00

   [upbeat music] Uh, while people get settled in, I'll do a quick survey, I guess.

2. 0:17

   Yeah, feel free. Go take my seat. Um, raise of hands, how many people use Google Photos in any way? Are you familiar with the application? Cool. How many people have used the editing feature from Google Photos?

3. 0:27

   Cool. A few less, that's fine. How many people have used, like, DALL-E or any of these new generative image editors? Okay, not that many. Cool. Okay. Well, my name is Kelvin.

4. 0:36

   Uh, I am an engineer on the Google Photos editing team. Really quickly about Google Photos, I think most people are familiar with it. We are the home for your memories.

5. 0:44

   We offer auto backup. When we first launched, the product really was built with machine learning in mind, right? We wanna use ML to make your life easier. Um, so if, for example, you, if you back up your photos with us, we index your photo, we run OCR, so you can do really powerful machine learning search, so you

6. 1:01

   don't have to organize your photos anymore. You go on a trip, you come back home, the photos get backed up. We go, "Hey, you went on a trip. Here's an album of your photos from that trip."

7. 1:08

   Um, you can just search for stuff like receipts from this restaurant, and we'll find it for you. We're all taking way too many photos nowadays, way too many video.

8. 1:15

   Nobody's got time to sit down and organize any of it, so let us do that for you.

9. 1:20

   We have about one point five billion monthly active users, and we do hundreds of millions of edits per month across our clients.

10. 1:29

    The team I work on is called the compu- computational photography team. I just call it the editing team. It's easier to say. Uh, it was started in twenty eighteen.

11. 1:37

    We had a pretty basic image editor at the time, but we wanted to really focus on this idea of using the compute on your phone, right? It doesn't have a great sensor, but it does have a lot of compute, right?

12. 1:48

    And we wanna use that to make really great image edits for any images. Doesn't have to be captured on the phone. It could be old devices too. And the idea is with computational photography, so for example, you have, like, this photo on the left that is kind of taken at sunset, and the subject is very dark.

13. 2:05

    Traditionally, with DSLRs, what you would do is you would capture multiple images at different exposure level, and then you combine them into the right image, which is called a HDR image.

14. 2:13

    But that's a lot of work. You need a tripod, you need to [chuckles] know how to do this, you need to know how to use Photoshop. With machine learning and compute, we can just do it with one image.

15. 2:22

    So capture anywhere you want, bring it to Google Photos. We run machine learning. We generate this photo. We kinda re-bring in all the brightness and variance in the image.

16. 2:32

    And the reason we're doing this at Google with the Photos team is we are ab- able to vertically integrate. We control the hardware from Pixel, right? We're able to use stuff like Edge TPU to really do accelerated compute.

17. 2:43

    But we also have internal research. Um, this is twenty eighteen. Back then, I think Hugging Face was founded, but it was not a place where you can just go on Hugging Face, find a model that suits your need, and then go off and build an application with it.

18. 2:54

    Like, you can do that now, which is really great, but you couldn't then. And we're able to work with our researchers in Google that are experts in computer vision and machine learning to build the feature from the ground up, right?

19. 3:05

    Um, like, "Hey, can you iterate on the model?" We'll iterate on the application. Let's keep go back and forth until we hit something really good that's easy to use.

20. 3:13

    I'll really briefly talk about our tech stack. We have three main clients, Android, iOS, web. Those are all pretty natural to each of those clients. My team also owns this shared C++ library that does all the model inference, and it's all on-device.

21. 3:26

    Um, we integrate across all the clients. We integrate with our research partners, um, and then we run inference on-device using, um, TensorFlow Lite, which is now called LiteRT.

22. 3:40

    I'll briefly show off some of the editing features, highlights that we built from twenty eighteen for the next couple years. The first one is this post-capture segmentation, right? You capture a photo of a portrait.

23. 3:51

    Oh, you decided, "I actually want more bokeh in the background. I didn't have a lens that could do bokeh in that environment." We're able to just do it for you after the fact.

24. 3:59

    Another one, you know, you capture a nice portrait. The lighting's not perfect. Maybe the sun's in your eyes. It's kind of, like, making you washed out, or it's, like, overcast, and you have shadows on your face.

25. 4:09

    Also, able to fix that after the fact. You know, you want a lighting on your left side, lighting on your right side, no problem. We'll do it for you.

26. 4:16

    Another one, this is pretty popular on one of the Pixel launches, is Magic Eraser, right? You might be at a popular spot. There are a lot of tourists or distractors in the background.

27. 4:24

    You want a really clean photo of just the core subjects. Um, no problem. Don't worry. Have... Don't have to wait for the scene to clean up or anything like that.

28. 4:32

    Just take the photo. We can auto-detect, like, what are the subjects you care about, and then just clean the background for you and inpaint it for you and get something really beautiful that you can print off or show off to your, you know, close friends or family members.

29. 4:47

    Diving briefly into, like, you know, the first big feature we launched, um, which is the post-capture segmentation. This is really simple, actually, relatively. It's a U-Net convolutional neural network.

30. 4:58

    This is, like, pretty traditional ML stuff in the computer vision space, right? And it's able to just focus on a specific use case where it works really well, which is single portrait, single subject portrait segmentation.

31. 5:09

    So you see below, we separate that into a foreground, which is the white, and the background. And this is where the strengths of ML really comes in. Like, you can do this with computer vision without using machine learning, right?

32. 5:21

    But it doesn't behave as well, and you need experts who are experts in that system to tune it. With machine learning, you just get better performance. You just build a benchmark, the dataset, and you do the training.

33. 5:32

    And it always returns a result. There's no error handling, like [chuckles] the beauties of models is it always works, right? You give it any input, it'll run, and it's like, "Here's the output," and that's nice.

34. 5:43

    There are some challenges. Models are really big. They're giant, um, static files of floats, right? In this case, this model was ten megabytes, which is a lot of code, and that's not something we can bundle in the application.

35. 5:53

    We're very sensitive to our APK size. So now we have to download the model after the fact. Then you have to do model management, IP protection, since we're doing on this on the user's device.

36. 6:03

    We have to make sure the model can't be extracted and used somewhere else.

37. 6:08

    Uh, also, I'm sure everyone here who, who's worked with AI, ML has to deal with evals. Like, how do you know this model does what you want? How do you know the next model is better and does better in what you want?

38. 6:19

    And that just takes a lot of time. You have to build a benchmark. You have to run the benchmark. Um, you have to make sure the benchmark actually reflects your u- real-world usage, 'cause if they separate, then the benchmark is useless, right?

39. 6:30

    I think of it as for traditional code, you have unit tests, and that's how you make sure you don't have regressions. The benchmark is the equivalent unit testing for your model, and you need to maintain it, and it takes time.

40. 6:41

    And then the pro is also a con. The model always returns a result. [chuckles] Like, it, uh, people deal with this LLM, so it's like, "Hey, tell me when you're not sure, [chuckles] so I can do something else."

41. 6:50

    And the LLM will be like, "What do you mean? I'm always sure," right? Even in this case, it'll always return you something. And for the example, even this segmentation, right?

42. 6:57

    The subject has really long hair, and you'll see the mask it produces is not perfect. It's not capturing her hair, the finer strands. So when you really do blur or you do some really sharp segmentation, you will notice that it's not perfect.

43. 7:09

    There's blurry edges and that sort of thing. And we can fix that post-model. We're able to run kind of more tr- image understanding traditionally and be like, "Hey, that's a hair.

44. 7:18

    Let's follow the strands of the hair," get a really perfect answer to that.

45. 7:23

    Uh, a couple years later, we launched Magic Eraser. This is in twenty twenty-one. At this point, we're really doing a system of models. You know, now people call it, like, orchestration in the LLM world.

46. 7:32

    But really, we have a few things here. We are detecting distractors. We are segmenting them. We are then running an inpainter on them. And then we have, um, custom GL rendering on your device to make all this seamless.

47. 7:44

    So visualize the mask, animate the mask away, bring in the inpainting area, so on and so forth.

48. 7:50

    More models, more things to con- Your system is now more complicated. You have larger models. They're getting to hundreds of megabytes now, even on device. Um, the failure cases are more obvious.

49. 8:00

    If you ask to inpaint something really large from the scene, very visible in the foreground, um, even a model will struggle with it, um, at least in twenty twenty-one.

50. 8:08

    As Paige mentioned, now we are able to do much better stuff. So quick summary of our learnings from those couple years, right? ML really does give you great capabilities that you wouldn't be able to do with traditional image understanding.

51. 8:21

    It's great that you can shape the, ship these features that are very easy to use. The models themselves have very consistent latency on a specific device, right? Once again, the models just does the same thing.

52. 8:31

    Doesn't matter the input. [chuckles] We're gonna run this number of flops. It's gonna go great. The device themselves on Android espec-especially have a huge variance. You know, the latest Samsung and Pixel really kinda comparable to certain laptops.

53. 8:44

    The really older phones, not at all comparable, so you have to deal with that. And then once again, going back, like, we really had this great relationship with our researchers, some of which is now in DeepMind, right?

54. 8:55

    Being able to talk to them early is like, "Hey, we have a use case. Our users want to be able to do certain things. What do you have that lets us build something on top?"

55. 9:03

    And we're able to go with them for years at a time, really, to see, like, "Oh, you made it better this year. It's not quite ready for our use case, but that's great.

56. 9:11

    Let's keep working on it." So that's all really good. The downside is, once again, the unpreddic- unpredictability of certain edge cases. Sometimes you'll look at two images and you're like, "They're the same to me."

57. 9:22

    You put one in the model, great result. Put another one in the model, terrible result. And you're like, "Why? Why?" And the researcher will be like, "Who knows? We should, you know, collect more data, [chuckles] train a new version of the model.

58. 9:32

    We'll let you know in a couple weeks or a month if it's fixed or not." That's just the nature of working with machine learning, right? No one can look at the system, go, "Oh, there's the bug.

59. 9:40

    Let me fix it. Let me push the fix. It'll be up." That's totally different from software engineering. So that means slower iteration. You gotta keep in mind, especially for us, we launch a Pixel.

60. 9:49

    We have hard deadlines. We're launching in two months. That means we get, what, two iterations of the model at best? Can you promise a launch? Can you go to your VP and go, "We are ready to launch"?

61. 9:58

    That's a totally different mindset from machine learning.

62. 10:03

    So what happened in twenty twenty-two, twenty twenty-three? What's the big excitement? Some of you probably know. Certain things launched. They covered it too with DALL-E and ChatGPT.

63. 10:14

    AI and AI, AI, AI, generative AI, generative AI, generative AI, AI is AI, AI, AI, AI, AI, AI is AI, AI, AI, AI. It uses AI to bring AI, AI, AI, and AI to AI.

64. 10:27

    AI and AI, AI, AI, generative AI, gen-

65. 10:30

    Yes. So that was just a snippet of I/O for that year. Uh, really, everyone's excited about AI. No more ML now. It's all AI. Larger models, more ambitious, more ambiguous.

66. 10:41

    Really, it's, that's, that's the world we live in now, right? So that's why... Well, that's not totally why, but, like, large reason why we d- decided to embark on building this new Magic Editor experience, which is now using the largest state-of-the-art models.

67. 10:53

    Um, but it also solves another problem. We've been building these great features, specific features, for years at this point, but discoverability is an issue. Like, the user still has to know, "Oh, in this c- this case, I wanna use Magic Eraser.

68. 11:05

    In this case, I want to blur the background." AI is great at going, "Hey, you know what? In this case, you should try using Magic Eraser." Like, it solves that, and we wanna combine both of them. [lip smack]

69. 11:16

    So from a product level, I think the previous speakers, like Paige, showed off, you know, this really amazing, fantastical ability, um, to generate, like, anything you want, like, something you can capture in the real world, some things you can't.

70. 11:28

    For photos, product-wise, we wanna be more grounded, right? We are the home for your memories. We don't want to generate something really weird that kind of would stand out if you were to share it with your friends or family.

71. 11:40

    We obviously will have a prompt. Everything has a prompt now. [chuckles] Um, but we want prompt to be more specific. Like, we want, still have the ability for users to select and visualize what they're talking about, so you can interact with the model, kind of as, like, a co-editor or a co-pilot, so on.

72. 11:55

    And of course, we have to use the best model available 'cause we wanna be more ambitious and do more, like, what's possible at the edge. So we're no longer constrained to on-device.

73. 12:04

    That means some challenges. Now we have to worry about servers. Before, everything happens on one device, and there are a lot of advantages there. But there's no way back then, or even now, we can fit the best models on a, a mobile device specifically.

74. 12:18

    Two, the problem space is really big. Like, it's great to say w-we wanna build the best GenAI image editor, and you're like, "Great, what does that mean?" [chuckles] Right? LLMs can do a lot of things.

75. 12:28

    Editing images is a subset of what they can do, and then within that space, there's still a lot of specific things they can do. So you gotta narrow what you are actually tackling 'cause, like, image-- generative image editing is not a specific user problem.

76. 12:41

    The user doesn't go, "I want a generative image edit." They have a use case, and you gotta, like, br- meet them where their use case is. And I think the other speakers also talked about, like, trust and safety is a big one for this sort of thing, preventing deepfakes, being responsible, so on and so forth.

77. 12:56

    So client-server, um, this is actually new for us. I'm coming from a on-device local-first background. I think most people are using AI server side. So up until this point, I have never had to do server capacity planning.

78. 13:10

    The user brings the compute. It is great. I don't pay for power. I don't pay for compute. You know, a billion people wanna use it, five billion people wanna use it, great.

79. 13:19

    Nothing changes on our end. We push the same code. But now we have to worry about it, especially 'cause we use accelerated compute, TPUs, GPUs. You really have to do a lot of planning, and this takes a lot of time, right?

80. 13:30

    Second, latency cons- latency is a concern now. Before, we had zero network latency on device. Now you worry about network quality. Is the user in a remote spot? Is the data center overloaded and they are backed up?

81. 13:42

    Is the user really far from the data center and now you have to go, like, a round trip around the world, and that adds a ton of latency. And then the other thing is testing is now hard, right?

82. 13:51

    Before, our models were small enough that we could run our, um, tests, including the models. Now these models are way too big. We can't run them in our automated testing suite for our daily regressions.

83. 14:02

    Now we have to worry about, like, do we recapture their responses? Do we try and have a test server? None of these are really good options here.

84. 14:11

    I do wanna spend a lot of time talking about this ambiguous problem space. There's this old comic from xkcd. This is, like, I don't know how old, but old when it made sense.

85. 14:19

    Basically, it's like, hey, some problems in computer science are hard to explain why it's hard. You know, the first one's like, let the-- Is the user in a national park?

86. 14:27

    No problem. GPS. That's been solved problem. Like, easy. Anyone can do it now. And then it's like, check the-- whether the photo is of a bird. And back then, that'd be like, "I need a computer vision researcher team.

87. 14:37

    Give me a couple PhDs. I'll get back to you." Now this is a solved problem, so the world has changed, right? And that's the power of GenAI, which is great.

88. 14:46

    Like, I love going to hackathons now 'cause anyone can sit down at a hackathon and be like, "You know what? I have this idea, and I can build it.

89. 14:52

    It will definitely work in some use case," and that's great. And when-- So when your PM goes, "I wanna solve this problem 'cause... And I used Gemini or ChatGPT, and it worked in this case, that means we can build it."

90. 15:04

    As engineer, you go like, "Did it work 5% of the time? Did it work 10% of the time? [chuckles] Did it work 50% of the time or 80% of the time?"

91. 15:10

    Those are-- It all worked in some case, but those are vastly different things. You cannot ship a product that is 5% reliable. If it's 50% reliable and you can work with researchers to get it to 80%, then let's talk.

92. 15:22

    But you really do wanna constrain the problem you're solving. I think someone talked about yesterday the comment that, like, prompt is a bug, not a feature, and I agree with that, right?

93. 15:32

    You want it to be easy to use. A user doesn't wanna look at a prompt and think about what they want in a very detailed way. Paige talked about, like, prompt editing, and we do some of that too.

94. 15:44

    The user doesn't wanna write a whole paragraph on their phone. We wanna extract their intent and go, "Great, we think we know what you want. Let us give you what you actually want, not what you say you want," right?

95. 15:56

    And then, once again, anything is possible means nothing is out of scope, which means you cannot design an engineering system around it. A system, by definition, has constraints.

96. 16:07

    So really what we focus on is reducing ambiguity across all of our functions. So product goes, "Hey, talk to users. What are the big demands they have?" Right? A-and they identified a few things here.

97. 16:18

    One, the ability to move things within the image, to relocate, right, seamlessly.

98. 16:26

    Another, reimagining some of the scenes. So this is, like, the left side is the real part. It was a very gray sky, kind of boring. Reimagine the background to be something more exciting.

99. 16:37

    And then we had eraser before, but now we can do better erase 'cause we're no longer constrained to the device capabilities. So here it's not just erasing the drink itself, but also the reflection of the drink, so it actually feels more natural.

100. 16:48

     And then we work with research to highlight those or train specific models for those use cases so they're more efficient, more accurate, um, easier to use. And then we build the UX and the software engineering on top to guide users through those cases so they work really, really well, so you can rely on them.

101. 17:04

     And then for the other use cases, we kind of just take the advantage of LLMs, which tend to hallucinate. But in our case, we'll just give you multiple responses.

102. 17:13

     The hallucination is a feature. The good thing is we work in a creative space. There is no correct. It's not like code. There's no compilers, right? It's like, if you see something you like out of these choices, great.

103. 17:23

     Take it. Go with it. Trust and safety, obviously this will never be perfect. This is not a world where we just go, "This is the correct answer 100% of the time."

104. 17:31

     You wanna aim for certain precision or recall, right? And you have to manage the expectations of your users, of journalists, of media, of, you know, your internal stakeholders to be like, "We are doing what we can.

105. 17:43

     We're preventing the really bad cases," but this is something we're gonna always keep going on. It's ambiguous. The language itself is ambiguous. The prompt could be like, "The view from the mountain was sick.

106. 17:53

     You should make it look like that." What does sick mean? Will the LLM make sure it interprets sick in the way they meant and not in the they were unhealthy way?

107. 18:03

     Like, that's just how human language works, and that's why we have code.

108. 18:08

     So my learnings on working with AI, once again, is like to me, AI engineering is just software engineering, but with machine learning or ML on top. And to me, ML is this great power, but it also adds a lot of randomness into your system.

109. 18:21

     And as engineers, our job is to reduce that randomness and kind of bring things back to a deterministic, repeatable way, so you can actually have repeatable outcomes that, like, provides value to your users in some way, right?

110. 18:34

     So build evals, use your evals, make your evals faster to run, and then once you have a useful value in production, go from a large model that can probably do way too much and is doing way too much compute for your use case, and replace with a smaller model, faster model, more efficient model, whether that's through model

111. 18:52

     distillation or you find a more efficient model, or you can even replace it with traditional engineering without ML. All great stuff. The main point is to be able to move faster, right?

112. 19:01

     Fast iteration means more tries, means you get better improvements in your product.

113. 19:08

     And then for what's next for us, we are-- we announced last week at the Google Photos 10-year anniversary, we're rebuilding the editor from the ground up to be AI first.

114. 19:17

     So once again, really fulfilling that vision of we are meeting you where you are. You can tap on your image, we'll surface the relevant edits, you can kind of adjust where they are, and you can use AI as part of that tool if you want.

115. 19:28

     But-- or the AI can use deterministic tools to do better edits.

116. 19:34

     Uh, I'm almost at time. This is more my personal view. Like, where will we be in two years? I don't know. You know, you talk to different researchers, they have different views on AI progress.

117. 19:43

     I think Paige talked about, you know, like the Gemma Nano on-device model now that was just released. Is this as good as the Pro model of the last version of Gemini?

118. 19:52

     I hear that too. That's a very exciting world to me, to bring things back on device. Or who knows, maybe Gemini 4 and these things keep increasing, and the Nano version is one step behind.

119. 20:02

     That's an interesting part too. Or maybe we have no progress. But either way, being able to iterate quickly and having really good benchmarks so that you can know what to change is always important, so we should just focus on that.

120. 20:13

     And I think that's my time. If you wanna contact me, that's my contact, and I'll be here afterwards to talk. [clapping]

121. 20:21

     Thanks, Kelvin. [outro jingle]