AI Engineer World's Fair 2025
Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App
Read the talk
Magic Editor Under the Hood: From Portrait Masks to Generative Editing
A dark sunset portrait captures the promise of computational photography. Delivering that promise at Google Photos scale requires constrained tasks, maintained evaluations and careful control of model uncertainty.
From a talk by Kelvin Ma
Before you start: Basic familiarity with image pixels, machine-learning inference and client/server applications will help; no computer-vision background is required.
Make the photo useful after capture
Can a photo app find a restaurant receipt without making you organize your pictures first? Google Photos was built around that kind of problem: automatic backup feeds image indexing, OCR and machine-learning search. A trip can become an automatically assembled album; a receipt can become searchable. Kelvin Ma, an engineer on the editing team, reports about 1.5 billion monthly active users and hundreds of millions of edits per month across clients.
The computational photography team, formed in 2018, brought the same approach to editing. A phone may have a limited sensor, but it has substantial compute—and an editor can apply that compute to pictures from older devices or other cameras, too. Consider the sunset portrait: the background is bright, but the subject is dark. Traditional DSLR HDR combines photographs taken at different exposures, potentially requiring a tripod and manual assembly in Photoshop. Ma’s demonstration instead enhances one existing image, restoring brightness and vibrance after capture.
Google could develop this through vertical integration: Pixel hardware and Edge TPU acceleration on one side, internal computer-vision researchers on the other. In 2018, Ma recalls, developers could not browse Hugging Face for a suitable model as readily as they can today. Researchers and application engineers instead iterated together, improving the model and the interaction until the feature became useful and easy to operate.
The original stack paired native Android, iOS and web clients with a shared C++ inference library. Model inference ran on-device through TensorFlow Lite, renamed LiteRT. This local execution model shaped both the features the team could build and the operational problems it had to solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with specific photographic problems
The early editing tools each addressed a recognizable defect in a photograph:
- Background blur: Post-capture portrait segmentation lets the editor add bokeh even when the photographer lacked a lens that could produce it at capture time.
- Portrait lighting: Relighting can address a washed-out face or unwanted shadows, including changing the apparent direction of illumination.
- Background distractions: Magic Eraser identifies important subjects and unwanted background objects, removes the distractions and inpaints the exposed regions. The photographer can take the picture without waiting for tourists to leave the scene.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A mask is an output, not a guarantee
The first major feature, post-capture segmentation, used a U-Net convolutional neural network scoped to single-subject portraits. Its output separates foreground from background, with the subject represented in white on the mask. Traditional computer vision could also attempt this separation, but it required specialists to tune the system. Machine learning shifted much of that work toward assembling a dataset, defining a benchmark and training against it.
A mask gives downstream image processing a concrete interface. For background blur, the following C++ composition keeps the original pixel where the foreground mask is white and uses an already-blurred image where it is black. Intermediate mask values blend the two at boundaries:
cpp
#include <algorithm>
#include <stdexcept>
#include <vector>
struct Rgb {
float r, g, b;
};
std::vector<Rgb> compositePortrait(
const std::vector<Rgb>& original,
const std::vector<Rgb>& blurred,
const std::vector<float>& foregroundMask) {
if (original.size() != blurred.size() ||
original.size() != foregroundMask.size()) {
throw std::invalid_argument("Image and mask sizes must match");
}
std::vector<Rgb> result(original.size());
for (std::size_t i = 0; i < result.size(); ++i) {
const float a = std::clamp(foregroundMask[i], 0.0f, 1.0f);
result[i] = {
a * original[i].r + (1.0f - a) * blurred[i].r,
a * original[i].g + (1.0f - a) * blurred[i].g,
a * original[i].b + (1.0f - a) * blurred[i].b
};
}
return result;
}
This makes mask quality consequential: pixels assigned to the background receive the background treatment, even when they belong to the subject.
A model returning an output for every input simplifies the application’s control flow, but shipping the model introduces another set of concerns. Ma reports that the segmentation model was 10 MB—already too large to bundle under the team’s APK-size constraints. Downloading it after installation meant managing model availability and versions, as well as protecting the on-device model against extraction and reuse.
Benchmarks are the model’s regression tests. They need maintenance just as code tests do: the team must run them, compare successive models and ensure that the dataset still represents real-world usage. A benchmark that drifts away from actual photographs stops answering the question the product team needs answered.
The apparent advantage of always returning a result is also a failure mode. In the portrait example, the mask misses fine strands of long hair. Applying a strong blur or sharp separation then exposes imperfect edges. Ma describes correcting this after inference with traditional image understanding that follows the hair strands. The useful editing system therefore includes both a learned segmentation stage and conventional processing around it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Magic Eraser becomes a system of models
By the 2021 launch of Magic Eraser, the feature required several coordinated stages:
- Detect distracting objects.
- Segment those objects to identify the regions to remove.
- Inpaint the selected regions with replacement image content.
- Use custom on-device GL rendering to display the masks, animate their removal and reveal the inpainted areas.
The visible interaction depended on the rendering system as well as the model outputs.
Ma reports that on-device models had grown into the hundreds of megabytes by this stage. More models meant more system complexity, while more ambitious edits made failures easier to notice. In the 2021 implementation, removing a large, prominent foreground object could ask the inpainter to reconstruct more of the scene than it could handle convincingly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predictable inference, unpredictable fixes
These models made powerful edits easy to request, and their execution was relatively predictable. On a particular device, a model performed the same amount of computation regardless of the image, giving it consistent latency. Across Android devices, however, performance varied substantially: recent Samsung and Pixel phones could resemble some laptops in capability, while older phones could not.
Research partnerships helped the team develop capabilities over years. Engineers could bring a user need to researchers before a model was ready, follow successive improvements and keep refining the application around them. But an individual failure was rarely as straightforward to repair as a conventional software bug. Two images could look nearly identical to a person and produce sharply different model results.
The response might be to collect more data and retrain, then wait weeks or a month to discover whether the problem improved. That uncertainty collides with hardware launch schedules. In Ma’s planning example, two months before a Pixel launch might allow at best two model iterations. A launch commitment has to account for that slow, uncertain feedback loop, rather than assuming an engineer can locate a bug and immediately ship a fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A generative editor for recognizable memories
The excitement around DALL-E and ChatGPT in 2022–2023 changed the ambition of image editing. Ma marks the transition with a Google I/O montage repeatedly saying “AI”: larger models brought more possibilities, but also a less clearly bounded problem. That enthusiasm helped motivate Magic Editor, though it was not the only reason to build it.
The existing tools had a discoverability problem. Users needed to know when to choose Magic Eraser, when to blur the background and where to find each operation. AI could help identify the relevant edit instead of requiring the user to understand the tool collection first.
The product still needed to fit a library of personal memories. Rather than emphasizing fantastical images, Photos aimed for edits that would remain plausible when shared with friends and family. Specific prompts combined with visible selection would let users show the model what they meant and interact with it as a co-editor. Supporting more ambitious edits with the best available models also meant leaving the original on-device constraint.
That introduced servers, but it also exposed a product-definition problem: generative image editing is not a specific user need. Even within image editing, an unrestricted model can attempt many different tasks. The team had to choose which needs to serve, while making deepfake prevention and responsible use part of the requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Server inference changes the operating costs
In the local-first system, users supplied inference compute and power. More people using the editor did not require the team to provision more inference servers. Moving to accelerated server compute introduced TPU and GPU capacity planning, with substantial planning time. Network conditions and data-center location also became part of the editing experience.
| Concern | On-device editing | Server-backed editing |
|---|---|---|
| Capacity | User supplies compute | Provision TPUs and GPUs |
| Latency | No inference network trip | Network, distance and queueing |
| Regression testing | Include small models in tests | Large models exceed the daily suite’s limits |
A remote user, an overloaded data center or a long geographic round trip can now delay an edit. Testing becomes harder, too: Ma says the larger models could no longer run inside the daily automated regression suite. Replaying captured responses and using a test server were possible approaches, but he presents neither as a satisfactory settled solution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A successful demo does not define a shippable task
The old xkcd comic “Tasks” contrasts two requests that sound similarly simple: determine whether someone is in a national park, and determine whether a photograph contains a bird. The first is a geographic lookup; historically, the second could require a computer-vision research team. Models have changed that boundary. A hackathon participant can now build an idea that works in at least some cases, and a product manager can demonstrate a desired behavior with Gemini or ChatGPT.
The engineering question is how often it works. Ma uses illustrative success rates of 5%, 10%, 50% and 80% to distinguish a compelling example from a dependable capability. He rejects shipping something that works only 5% of the time; a capability that works half the time and has a credible research path toward 80% is at least worth discussing. These are hypothetical planning examples, not measured Magic Editor results.
Constraining the task also reduces the burden on the user. Ma agrees with the formulation “prompt is a bug, not a feature”: someone editing a photograph on a phone should not need to write a detailed paragraph. Prompt editing and intent extraction can translate a small amount of input into a more useful request. But the system still needs boundaries. If anything is possible and nothing is out of scope, engineers have no stable contract around which to design it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn user intent into supported edits
User research narrowed the broad editing space to concrete demands:
- Relocate: Move an object within the image seamlessly.
- Reimagine: Replace a dull gray-sky background with a more interesting scene.
- Erase: Remove an unwanted object and the visual evidence that belongs to it. In Ma’s example, the stronger eraser removes both a drink and its reflection, producing a more natural result than removing the drink alone.
Those tasks give research and product engineering something specific to optimize. Researchers can emphasize or train models for the chosen cases; application engineers can build guided interactions that make those cases easier to use reliably. For other requests, the editor can offer multiple generated alternatives. Creative editing has no single correct image, so variation that would be a defect in a factual answer can become useful choice: the user selects a result they like.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Safety has to handle ambiguous intent
Creative flexibility does not remove the need for trust and safety. Ma describes a continuing effort governed by precision and recall targets, rather than a promise that every decision will be correct. The team must prevent severe cases while setting realistic expectations with users, journalists and internal stakeholders.
Language itself complicates that work. A user might say the view from a mountain was “sick” and ask the editor to make it look that way. Does the word mean impressive, or does it refer to illness? The intended edit depends on interpreting that ambiguity correctly. Restricting the interaction can reduce uncertainty, but free-form language keeps introducing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use production learning to simplify the system
Ma describes AI engineering as software engineering with machine learning adding randomness to the system. The engineering responsibility is to reduce that randomness enough to produce repeatable outcomes that users value. The goal is dependable behavior, not the largest possible model.
That suggests a practical progression:
- Build evaluations, use them and make them faster to run.
- Establish that the capability provides useful value in production.
- Replace an oversized model with a smaller, faster or more efficient implementation where the evaluations support it.
The replacement might come from distillation, a different model or traditional engineering without machine learning. Each can shorten the iteration loop, giving the team more attempts to improve the product.
The next editor design extends this approach to the interaction itself. Google’s May 28, 2025 tenth-anniversary announcement introduced a rebuilt, AI-first editor; Ma describes it as newly announced at the time of the talk. Users would tap an image to surface relevant edits and adjust them. They could invoke AI as a tool, while AI could also invoke deterministic editing tools to produce a better result.
Where that compute will run remains uncertain. In his personal two-year outlook, Ma asks whether Gemini Nano has reached the capability of a previous Gemini Pro, citing something he has heard rather than a measured comparison. That possibility makes a return to on-device inference exciting to him. Alternatively, larger models—including a hypothetical Gemini 4—might keep advancing while smaller models remain behind, or progress might stall. Across those futures, fast iteration and strong benchmarks still tell the team what to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The May 2025 announcement explains suggested edits, selection-based tools and the planned redesigned-editor rollout.
Google’s original preview illustrates subject relocation, background changes and filling missing image content.
The launch explanation covers distraction detection, user selection and replacement-pixel generation.
Ronneberger, Fischer and Brox introduce an encoder-decoder architecture for precise image segmentation.
The comic contrasts a straightforward geographic lookup with the historically difficult task of recognizing a bird.
Further reading
Google Research explains Ultra HDR gain maps and a lightweight model that reconstructs them after image edits.
A May 2025 implementation guide explains task-specific adaptation, evaluation and Kotlin usage for on-device text and image-description features.
Updates since the talk
Current documentation for deploying machine-learning models on devices with LiteRT.
Read the complete timestamped transcript
- 0:00
[upbeat music] Uh, while people get settled in, I'll do a quick survey, I guess.
- 0:17
Yeah, feel free. Go take my seat. Um, raise of hands, how many people use Google Photos in any way? Are you familiar with the application? Cool. How many people have used the editing feature from Google Photos?
- 0:27
Cool. A few less, that's fine. How many people have used, like, DALL-E or any of these new generative image editors? Okay, not that many. Cool. Okay. Well, my name is Kelvin.
- 0:36
Uh, I am an engineer on the Google Photos editing team. Really quickly about Google Photos, I think most people are familiar with it. We are the home for your memories.
- 0:44
We offer auto backup. When we first launched, the product really was built with machine learning in mind, right? We wanna use ML to make your life easier. Um, so if, for example, you, if you back up your photos with us, we index your photo, we run OCR, so you can do really powerful machine learning search, so you
- 1:01
don't have to organize your photos anymore. You go on a trip, you come back home, the photos get backed up. We go, "Hey, you went on a trip. Here's an album of your photos from that trip."
- 1:08
Um, you can just search for stuff like receipts from this restaurant, and we'll find it for you. We're all taking way too many photos nowadays, way too many video.
- 1:15
Nobody's got time to sit down and organize any of it, so let us do that for you.
- 1:20
We have about one point five billion monthly active users, and we do hundreds of millions of edits per month across our clients.
- 1:29
The team I work on is called the compu- computational photography team. I just call it the editing team. It's easier to say. Uh, it was started in twenty eighteen.
- 1:37
We had a pretty basic image editor at the time, but we wanted to really focus on this idea of using the compute on your phone, right? It doesn't have a great sensor, but it does have a lot of compute, right?
- 1:48
And we wanna use that to make really great image edits for any images. Doesn't have to be captured on the phone. It could be old devices too. And the idea is with computational photography, so for example, you have, like, this photo on the left that is kind of taken at sunset, and the subject is very dark.
- 2:05
Traditionally, with DSLRs, what you would do is you would capture multiple images at different exposure level, and then you combine them into the right image, which is called a HDR image.
- 2:13
But that's a lot of work. You need a tripod, you need to [chuckles] know how to do this, you need to know how to use Photoshop. With machine learning and compute, we can just do it with one image.
- 2:22
So capture anywhere you want, bring it to Google Photos. We run machine learning. We generate this photo. We kinda re-bring in all the brightness and variance in the image.
- 2:32
And the reason we're doing this at Google with the Photos team is we are ab- able to vertically integrate. We control the hardware from Pixel, right? We're able to use stuff like Edge TPU to really do accelerated compute.
- 2:43
But we also have internal research. Um, this is twenty eighteen. Back then, I think Hugging Face was founded, but it was not a place where you can just go on Hugging Face, find a model that suits your need, and then go off and build an application with it.
- 2:54
Like, you can do that now, which is really great, but you couldn't then. And we're able to work with our researchers in Google that are experts in computer vision and machine learning to build the feature from the ground up, right?
- 3:05
Um, like, "Hey, can you iterate on the model?" We'll iterate on the application. Let's keep go back and forth until we hit something really good that's easy to use.
- 3:13
I'll really briefly talk about our tech stack. We have three main clients, Android, iOS, web. Those are all pretty natural to each of those clients. My team also owns this shared C++ library that does all the model inference, and it's all on-device.
- 3:26
Um, we integrate across all the clients. We integrate with our research partners, um, and then we run inference on-device using, um, TensorFlow Lite, which is now called LiteRT.
- 3:40
I'll briefly show off some of the editing features, highlights that we built from twenty eighteen for the next couple years. The first one is this post-capture segmentation, right? You capture a photo of a portrait.
- 3:51
Oh, you decided, "I actually want more bokeh in the background. I didn't have a lens that could do bokeh in that environment." We're able to just do it for you after the fact.
- 3:59
Another one, you know, you capture a nice portrait. The lighting's not perfect. Maybe the sun's in your eyes. It's kind of, like, making you washed out, or it's, like, overcast, and you have shadows on your face.
- 4:09
Also, able to fix that after the fact. You know, you want a lighting on your left side, lighting on your right side, no problem. We'll do it for you.
- 4:16
Another one, this is pretty popular on one of the Pixel launches, is Magic Eraser, right? You might be at a popular spot. There are a lot of tourists or distractors in the background.
- 4:24
You want a really clean photo of just the core subjects. Um, no problem. Don't worry. Have... Don't have to wait for the scene to clean up or anything like that.
- 4:32
Just take the photo. We can auto-detect, like, what are the subjects you care about, and then just clean the background for you and inpaint it for you and get something really beautiful that you can print off or show off to your, you know, close friends or family members.
- 4:47
Diving briefly into, like, you know, the first big feature we launched, um, which is the post-capture segmentation. This is really simple, actually, relatively. It's a U-Net convolutional neural network.
- 4:58
This is, like, pretty traditional ML stuff in the computer vision space, right? And it's able to just focus on a specific use case where it works really well, which is single portrait, single subject portrait segmentation.
- 5:09
So you see below, we separate that into a foreground, which is the white, and the background. And this is where the strengths of ML really comes in. Like, you can do this with computer vision without using machine learning, right?
- 5:21
But it doesn't behave as well, and you need experts who are experts in that system to tune it. With machine learning, you just get better performance. You just build a benchmark, the dataset, and you do the training.
- 5:32
And it always returns a result. There's no error handling, like [chuckles] the beauties of models is it always works, right? You give it any input, it'll run, and it's like, "Here's the output," and that's nice.
- 5:43
There are some challenges. Models are really big. They're giant, um, static files of floats, right? In this case, this model was ten megabytes, which is a lot of code, and that's not something we can bundle in the application.
- 5:53
We're very sensitive to our APK size. So now we have to download the model after the fact. Then you have to do model management, IP protection, since we're doing on this on the user's device.
- 6:03
We have to make sure the model can't be extracted and used somewhere else.
- 6:08
Uh, also, I'm sure everyone here who, who's worked with AI, ML has to deal with evals. Like, how do you know this model does what you want? How do you know the next model is better and does better in what you want?
- 6:19
And that just takes a lot of time. You have to build a benchmark. You have to run the benchmark. Um, you have to make sure the benchmark actually reflects your u- real-world usage, 'cause if they separate, then the benchmark is useless, right?
- 6:30
I think of it as for traditional code, you have unit tests, and that's how you make sure you don't have regressions. The benchmark is the equivalent unit testing for your model, and you need to maintain it, and it takes time.
- 6:41
And then the pro is also a con. The model always returns a result. [chuckles] Like, it, uh, people deal with this LLM, so it's like, "Hey, tell me when you're not sure, [chuckles] so I can do something else."
- 6:50
And the LLM will be like, "What do you mean? I'm always sure," right? Even in this case, it'll always return you something. And for the example, even this segmentation, right?
- 6:57
The subject has really long hair, and you'll see the mask it produces is not perfect. It's not capturing her hair, the finer strands. So when you really do blur or you do some really sharp segmentation, you will notice that it's not perfect.
- 7:09
There's blurry edges and that sort of thing. And we can fix that post-model. We're able to run kind of more tr- image understanding traditionally and be like, "Hey, that's a hair.
- 7:18
Let's follow the strands of the hair," get a really perfect answer to that.
- 7:23
Uh, a couple years later, we launched Magic Eraser. This is in twenty twenty-one. At this point, we're really doing a system of models. You know, now people call it, like, orchestration in the LLM world.
- 7:32
But really, we have a few things here. We are detecting distractors. We are segmenting them. We are then running an inpainter on them. And then we have, um, custom GL rendering on your device to make all this seamless.
- 7:44
So visualize the mask, animate the mask away, bring in the inpainting area, so on and so forth.
- 7:50
More models, more things to con- Your system is now more complicated. You have larger models. They're getting to hundreds of megabytes now, even on device. Um, the failure cases are more obvious.
- 8:00
If you ask to inpaint something really large from the scene, very visible in the foreground, um, even a model will struggle with it, um, at least in twenty twenty-one.
- 8:08
As Paige mentioned, now we are able to do much better stuff. So quick summary of our learnings from those couple years, right? ML really does give you great capabilities that you wouldn't be able to do with traditional image understanding.
- 8:21
It's great that you can shape the, ship these features that are very easy to use. The models themselves have very consistent latency on a specific device, right? Once again, the models just does the same thing.
- 8:31
Doesn't matter the input. [chuckles] We're gonna run this number of flops. It's gonna go great. The device themselves on Android espec-especially have a huge variance. You know, the latest Samsung and Pixel really kinda comparable to certain laptops.
- 8:44
The really older phones, not at all comparable, so you have to deal with that. And then once again, going back, like, we really had this great relationship with our researchers, some of which is now in DeepMind, right?
- 8:55
Being able to talk to them early is like, "Hey, we have a use case. Our users want to be able to do certain things. What do you have that lets us build something on top?"
- 9:03
And we're able to go with them for years at a time, really, to see, like, "Oh, you made it better this year. It's not quite ready for our use case, but that's great.
- 9:11
Let's keep working on it." So that's all really good. The downside is, once again, the unpreddic- unpredictability of certain edge cases. Sometimes you'll look at two images and you're like, "They're the same to me."
- 9:22
You put one in the model, great result. Put another one in the model, terrible result. And you're like, "Why? Why?" And the researcher will be like, "Who knows? We should, you know, collect more data, [chuckles] train a new version of the model.
- 9:32
We'll let you know in a couple weeks or a month if it's fixed or not." That's just the nature of working with machine learning, right? No one can look at the system, go, "Oh, there's the bug.
- 9:40
Let me fix it. Let me push the fix. It'll be up." That's totally different from software engineering. So that means slower iteration. You gotta keep in mind, especially for us, we launch a Pixel.
- 9:49
We have hard deadlines. We're launching in two months. That means we get, what, two iterations of the model at best? Can you promise a launch? Can you go to your VP and go, "We are ready to launch"?
- 9:58
That's a totally different mindset from machine learning.
- 10:03
So what happened in twenty twenty-two, twenty twenty-three? What's the big excitement? Some of you probably know. Certain things launched. They covered it too with DALL-E and ChatGPT.
- 10:14
AI and AI, AI, AI, generative AI, generative AI, generative AI, AI is AI, AI, AI, AI, AI, AI is AI, AI, AI, AI. It uses AI to bring AI, AI, AI, and AI to AI.
- 10:27
AI and AI, AI, AI, generative AI, gen-
- 10:30
Yes. So that was just a snippet of I/O for that year. Uh, really, everyone's excited about AI. No more ML now. It's all AI. Larger models, more ambitious, more ambiguous.
- 10:41
Really, it's, that's, that's the world we live in now, right? So that's why... Well, that's not totally why, but, like, large reason why we d- decided to embark on building this new Magic Editor experience, which is now using the largest state-of-the-art models.
- 10:53
Um, but it also solves another problem. We've been building these great features, specific features, for years at this point, but discoverability is an issue. Like, the user still has to know, "Oh, in this c- this case, I wanna use Magic Eraser.
- 11:05
In this case, I want to blur the background." AI is great at going, "Hey, you know what? In this case, you should try using Magic Eraser." Like, it solves that, and we wanna combine both of them. [lip smack]
- 11:16
So from a product level, I think the previous speakers, like Paige, showed off, you know, this really amazing, fantastical ability, um, to generate, like, anything you want, like, something you can capture in the real world, some things you can't.
- 11:28
For photos, product-wise, we wanna be more grounded, right? We are the home for your memories. We don't want to generate something really weird that kind of would stand out if you were to share it with your friends or family.
- 11:40
We obviously will have a prompt. Everything has a prompt now. [chuckles] Um, but we want prompt to be more specific. Like, we want, still have the ability for users to select and visualize what they're talking about, so you can interact with the model, kind of as, like, a co-editor or a co-pilot, so on.
- 11:55
And of course, we have to use the best model available 'cause we wanna be more ambitious and do more, like, what's possible at the edge. So we're no longer constrained to on-device.
- 12:04
That means some challenges. Now we have to worry about servers. Before, everything happens on one device, and there are a lot of advantages there. But there's no way back then, or even now, we can fit the best models on a, a mobile device specifically.
- 12:18
Two, the problem space is really big. Like, it's great to say w-we wanna build the best GenAI image editor, and you're like, "Great, what does that mean?" [chuckles] Right? LLMs can do a lot of things.
- 12:28
Editing images is a subset of what they can do, and then within that space, there's still a lot of specific things they can do. So you gotta narrow what you are actually tackling 'cause, like, image-- generative image editing is not a specific user problem.
- 12:41
The user doesn't go, "I want a generative image edit." They have a use case, and you gotta, like, br- meet them where their use case is. And I think the other speakers also talked about, like, trust and safety is a big one for this sort of thing, preventing deepfakes, being responsible, so on and so forth.
- 12:56
So client-server, um, this is actually new for us. I'm coming from a on-device local-first background. I think most people are using AI server side. So up until this point, I have never had to do server capacity planning.
- 13:10
The user brings the compute. It is great. I don't pay for power. I don't pay for compute. You know, a billion people wanna use it, five billion people wanna use it, great.
- 13:19
Nothing changes on our end. We push the same code. But now we have to worry about it, especially 'cause we use accelerated compute, TPUs, GPUs. You really have to do a lot of planning, and this takes a lot of time, right?
- 13:30
Second, latency cons- latency is a concern now. Before, we had zero network latency on device. Now you worry about network quality. Is the user in a remote spot? Is the data center overloaded and they are backed up?
- 13:42
Is the user really far from the data center and now you have to go, like, a round trip around the world, and that adds a ton of latency. And then the other thing is testing is now hard, right?
- 13:51
Before, our models were small enough that we could run our, um, tests, including the models. Now these models are way too big. We can't run them in our automated testing suite for our daily regressions.
- 14:02
Now we have to worry about, like, do we recapture their responses? Do we try and have a test server? None of these are really good options here.
- 14:11
I do wanna spend a lot of time talking about this ambiguous problem space. There's this old comic from xkcd. This is, like, I don't know how old, but old when it made sense.
- 14:19
Basically, it's like, hey, some problems in computer science are hard to explain why it's hard. You know, the first one's like, let the-- Is the user in a national park?
- 14:27
No problem. GPS. That's been solved problem. Like, easy. Anyone can do it now. And then it's like, check the-- whether the photo is of a bird. And back then, that'd be like, "I need a computer vision researcher team.
- 14:37
Give me a couple PhDs. I'll get back to you." Now this is a solved problem, so the world has changed, right? And that's the power of GenAI, which is great.
- 14:46
Like, I love going to hackathons now 'cause anyone can sit down at a hackathon and be like, "You know what? I have this idea, and I can build it.
- 14:52
It will definitely work in some use case," and that's great. And when-- So when your PM goes, "I wanna solve this problem 'cause... And I used Gemini or ChatGPT, and it worked in this case, that means we can build it."
- 15:04
As engineer, you go like, "Did it work 5% of the time? Did it work 10% of the time? [chuckles] Did it work 50% of the time or 80% of the time?"
- 15:10
Those are-- It all worked in some case, but those are vastly different things. You cannot ship a product that is 5% reliable. If it's 50% reliable and you can work with researchers to get it to 80%, then let's talk.
- 15:22
But you really do wanna constrain the problem you're solving. I think someone talked about yesterday the comment that, like, prompt is a bug, not a feature, and I agree with that, right?
- 15:32
You want it to be easy to use. A user doesn't wanna look at a prompt and think about what they want in a very detailed way. Paige talked about, like, prompt editing, and we do some of that too.
- 15:44
The user doesn't wanna write a whole paragraph on their phone. We wanna extract their intent and go, "Great, we think we know what you want. Let us give you what you actually want, not what you say you want," right?
- 15:56
And then, once again, anything is possible means nothing is out of scope, which means you cannot design an engineering system around it. A system, by definition, has constraints.
- 16:07
So really what we focus on is reducing ambiguity across all of our functions. So product goes, "Hey, talk to users. What are the big demands they have?" Right? A-and they identified a few things here.
- 16:18
One, the ability to move things within the image, to relocate, right, seamlessly.
- 16:26
Another, reimagining some of the scenes. So this is, like, the left side is the real part. It was a very gray sky, kind of boring. Reimagine the background to be something more exciting.
- 16:37
And then we had eraser before, but now we can do better erase 'cause we're no longer constrained to the device capabilities. So here it's not just erasing the drink itself, but also the reflection of the drink, so it actually feels more natural.
- 16:48
And then we work with research to highlight those or train specific models for those use cases so they're more efficient, more accurate, um, easier to use. And then we build the UX and the software engineering on top to guide users through those cases so they work really, really well, so you can rely on them.
- 17:04
And then for the other use cases, we kind of just take the advantage of LLMs, which tend to hallucinate. But in our case, we'll just give you multiple responses.
- 17:13
The hallucination is a feature. The good thing is we work in a creative space. There is no correct. It's not like code. There's no compilers, right? It's like, if you see something you like out of these choices, great.
- 17:23
Take it. Go with it. Trust and safety, obviously this will never be perfect. This is not a world where we just go, "This is the correct answer 100% of the time."
- 17:31
You wanna aim for certain precision or recall, right? And you have to manage the expectations of your users, of journalists, of media, of, you know, your internal stakeholders to be like, "We are doing what we can.
- 17:43
We're preventing the really bad cases," but this is something we're gonna always keep going on. It's ambiguous. The language itself is ambiguous. The prompt could be like, "The view from the mountain was sick.
- 17:53
You should make it look like that." What does sick mean? Will the LLM make sure it interprets sick in the way they meant and not in the they were unhealthy way?
- 18:03
Like, that's just how human language works, and that's why we have code.
- 18:08
So my learnings on working with AI, once again, is like to me, AI engineering is just software engineering, but with machine learning or ML on top. And to me, ML is this great power, but it also adds a lot of randomness into your system.
- 18:21
And as engineers, our job is to reduce that randomness and kind of bring things back to a deterministic, repeatable way, so you can actually have repeatable outcomes that, like, provides value to your users in some way, right?
- 18:34
So build evals, use your evals, make your evals faster to run, and then once you have a useful value in production, go from a large model that can probably do way too much and is doing way too much compute for your use case, and replace with a smaller model, faster model, more efficient model, whether that's through model
- 18:52
distillation or you find a more efficient model, or you can even replace it with traditional engineering without ML. All great stuff. The main point is to be able to move faster, right?
- 19:01
Fast iteration means more tries, means you get better improvements in your product.
- 19:08
And then for what's next for us, we are-- we announced last week at the Google Photos 10-year anniversary, we're rebuilding the editor from the ground up to be AI first.
- 19:17
So once again, really fulfilling that vision of we are meeting you where you are. You can tap on your image, we'll surface the relevant edits, you can kind of adjust where they are, and you can use AI as part of that tool if you want.
- 19:28
But-- or the AI can use deterministic tools to do better edits.
- 19:34
Uh, I'm almost at time. This is more my personal view. Like, where will we be in two years? I don't know. You know, you talk to different researchers, they have different views on AI progress.
- 19:43
I think Paige talked about, you know, like the Gemma Nano on-device model now that was just released. Is this as good as the Pro model of the last version of Gemini?
- 19:52
I hear that too. That's a very exciting world to me, to bring things back on device. Or who knows, maybe Gemini 4 and these things keep increasing, and the Nano version is one step behind.
- 20:02
That's an interesting part too. Or maybe we have no progress. But either way, being able to iterate quickly and having really good benchmarks so that you can know what to change is always important, so we should just focus on that.
- 20:13
And I think that's my time. If you wanna contact me, that's my contact, and I'll be here afterwards to talk. [clapping]
- 20:21
Thanks, Kelvin. [outro jingle]