AI Engineer World's Fair 2026
Building Closed-Loop Evals for a Multimodal Agent at Uber Scale
Read the talk
Building Closed-Loop Evals for a Multimodal Agent at Uber Scale
Improving food photos requires more than attractive edits: Uber’s agent system evaluates routing, preserves dish identity, and turns production feedback into benchmarked configuration updates.
From a talk by Soumya Gupta and Jai Chopra
Before you start: Familiarity with LLM agents, structured JSON output, and classification precision and recall will help you follow the evaluation design.
How do you improve a food photo without changing the food?
How can a delivery marketplace improve millions of food photos while keeping them representative of what customers will receive? The scale makes this an engineering problem as well as a photography problem. Jai Chopra reports an approximately $90 billion annual marketplace run rate, 20% year-on-year growth, operations in 10,000 cities, and millions of items added each month. These are the talk’s contextual figures, with no reference period specified; the marketplace run rate should not be read as Uber revenue. Chopra describes the delivery marketplace as comparable in size to Uber’s mobility business.
A photo is often a customer’s first impression of a merchant. It can determine whether someone keeps scrolling, opens an item, or adds it to the cart. Video is becoming more prominent too, expanding the visual content the marketplace must handle. But independent merchants often have photos that do not reflect the quality of their food. Their constraints are practical: time, photography know-how, and cost. Professional shoots are expensive, particularly when changing menus require another round of photography.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preserve authenticity and marketplace diversity
Better-looking images are useful only if customers trust them. Chopra describes consumer distrust of AI-generated imagery as a constraint on the product: an edit must remain faithful to the original food, preserve the merchant’s brand, and avoid making every restaurant look alike. A single editing prompt can collapse marketplace diversity, even if its individual outputs look polished.
The inputs also resist a uniform treatment. Across a global marketplace, photographs have different combinations of poor sharpness, off-center composition, weak colors, and other quality problems. User-generated content adds further variation. Enhancement therefore needs to be selective: improve what needs improvement while preserving what already works. The goals extend beyond individual images to marketplace-wide outcomes, avoiding gains that simply cannibalize other merchants, shipping safely, learning continuously, and keeping operating costs manageable.
There are two unhelpful extremes. A deterministic rules system provides control but becomes brittle across the long tail of images. An unconstrained agent has the creative freedom to generalize, but can violate the safety requirements that make an edit acceptable. Uber’s design gives agents room to act inside explicit guardrails. The architecture that follows is a simplified but representative account of the production system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route, edit, check—and record the whole execution
The execution path begins with image understanding and routing. A multimodal LLM describes the photograph and produces structured output. A router uses that assessment to choose between enhancement and skipping. Skipping keeps the original image. Enhancement sends the image to an editing agent, which receives feedback from a QA agent and can revise its work in a loop. If repeated attempts still fail, the system does not publish the edited image. A successful candidate proceeds to final post-processing and QA before menu publication.
Every stage is logged. In the representation Chopra describes, the agents’ records sit together in a flat JSON structure, making a complete case accessible to both engineering and product colleagues. The same records support aggregate analysis. An illustrative record for the skip path makes the decision and the absence of downstream work explicit:
json
{
"image_understanding": {
"assessment": "No enhancement needed"
},
"router": {
"decision": "skip"
},
"image_editing": null,
"editing_qa": null,
"publish_ready_qa": null,
"outcome": "keep_original"
}
The field names illustrate the flat arrangement rather than reproduce Uber’s schema. Its value is that someone investigating a case can see both what happened and which stages were never invoked.
Logging precedes optimization. Without execution records, there is little evidence with which to diagnose failures or build a learning loop. Chopra names Arize as Uber’s observability platform. The flat JSON arrangement described here is Uber’s execution representation, not a claim that Arize’s general trace model is flat.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the router against human judgments
The router combines the image with its text description and metadata. The model describes what it sees in structured form; that output is graded against a rubric with pass/fail criteria, producing the enhance-or-skip decision. This separates the visual assessment from the routing choice and gives the team a classification problem to evaluate. A confusion matrix exposes correct decisions and misses, with precision and recall measuring different aspects of the result.
More elaborate routing can choose an execution strategy as well as decide whether to edit.
| Routing problem | Evaluation structure | Trade-off |
|---|---|---|
| Enhance or skip | 2 × 2 confusion matrix | Which images need intervention? |
| Choose among multiple branches | n × n confusion matrix | Which branch fits each input? |
For example, a branch might send an image to a smaller, lower-latency model to reduce cost and waiting time at a possible quality cost. Each cell in the larger matrix then helps expose whether inputs reach the appropriate branch.
For the first release, Soumya Gupta treats human labels as the reference the model should match. The process starts with a representative dataset spanning geography, dish type, and image quality. Labelers receive objective guidelines intended to reduce subjective bias and noise. The team then runs the agent, compares its outputs with that golden dataset, and tunes until the release guardrails pass. For routing, the guardrail is recall: the system should not let bad images slip through.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A routing error can become a faithfulness error
The cheeseburger example shows the cost of unnecessary intervention. A high-quality photograph is incorrectly sent for enhancement. Gupta describes this as paying compute cost for no quality lift, while also exposing an already good photograph to degradation. Correctly deciding to leave an image alone is part of the product’s quality control.
The chicken-wing example exposes a different risk: the photograph shows six wings, while the dish description specifies eight pieces. Gupta presents the router’s approval as a recall miss. The downstream hazard is an editor trying to reconcile that mismatch by generating two additional wings. Those additions would match the text while violating faithfulness to the photograph. The example makes multimodal evaluation consequential: an image and its description can disagree, and an edit must not silently resolve that disagreement by inventing food. These long-tail failures remain possible after the offline model has passed its initial evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn production mismatches into benchmarked configurations
A static model cannot cover a changing production distribution indefinitely. Uber regularly samples production data and sends it to human labelers using the same guidelines as the offline benchmark. Agent-label mismatches go to an umbrella diagnoser agent, which localizes the problem and triggers autotuning. The tuned agent must pass the established golden-dataset metrics before it can ship; otherwise, the loop continues. Although routing is the example, Gupta describes this capability as applying to every component.
The diagnoser can write configuration and trigger the tuning pipeline without a person manually orchestrating those steps. That automation still depends on the human-labeled samples entering the loop. The distinction matters: human judgments establish the target, while configuration-driven machinery carries out diagnosis, optimization, and release evaluation.
Inside autotuning, the prompt optimizer divides its work between Reflect and Synthesize:
- Evaluate the target agent on unseen human-labeled samples, identifying matches and mismatches.
- Have Reflect examine the mismatches, filter noise, and identify systematic problems.
- Pass that feedback and the current agent configuration to Synthesize, which produces an updated configuration.
- Benchmark the updated agent. If it passes, register the new version in the agent configuration store.
- Let the next production run pick up the registered version.
This makes a proposed configuration distinct from a configuration eligible for production: the benchmark sits between them.
The automated loop also has observability on its guardrails and a quick rollback mechanism. Benchmarking controls promotion; monitoring and rollback address problems that become visible after promotion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate an image-specific edit, then feed back its failures
Once an image reaches enhancement, the system generates a prompt specifically for it. Prompt generation takes the description and the routing agent’s directives about what needs improvement. The resulting prompt drives image enhancement, followed by a multidimensional QA gate that examines properties such as plating, faithfulness, and colors.
A failed candidate returns QA feedback to prompt generation alongside the original inputs. The next attempt therefore has both the initial task and evidence about what went wrong. This is a bounded loop: within K iterations, the system either obtains a candidate that passes the editing gate or accepts a coverage hit and produces no accepted enhancement. It does not have to publish an edit merely because it spent compute generating one.
In the sweet potato fries example, the first edit fails QA for incorrect portion size and unrealistic plating; the second edit passes after receiving that feedback. Gupta calls the metric pass at K, describing it as the pass rate at the Kth iteration and expecting additional feedback to improve acceptance. The talk does not specify whether the aggregate rate is cumulative through K attempts or measured only at iteration K, so that definition must be made explicit before implementing the metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define better by comparing the original and the edit
Generation evaluation uses pairwise comparison: assess the input image beside the output and determine whether the output is better. That requires a product definition of better, not just an aesthetic preference. Uber aligns the rubric with product, design, policy, and legal. The detailed criteria are proprietary, but the disclosed dimensions include faithfulness, completeness, naturalness, and realism. The resulting judgment can be yes, no, or unsure.
Two examples separate adding unsupported content from losing existing content:
| Edit | Failed criterion | What changed |
|---|---|---|
| Adds shrimp | Faithfulness | Introduces food absent from the input |
| Removes sauce beneath sushi | Completeness | Deletes part of the original dish |
Both outputs may appear visually plausible. Comparing them with the original reveals why neither is an acceptable enhancement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Catch overcorrection, incoherent objects, and uncertainty
Feedback can also steer an editor toward an unhelpful solution. In one example, a creative first edit is rejected, and the next attempt overcorrects toward a generic ceramic bowl. Chopra describes this as reward hacking: the agent becomes overly conservative in response to QA. The pixels can change substantially without producing a meaningful improvement. An evaluator that rewards change alone would miss the distinction between a different image and a useful edit.
Another output has a plate covering the sauce. This is a failure of object coherence and physical plausibility, and it illustrates how limitations in the underlying image-editing models reach the applied product. Uber sometimes shares these failures with frontier model teams and works with them on the underlying problems. Application-level QA thus identifies both mistakes the agent configuration might address and weaknesses in the model doing the editing.
Uncertainty is itself a reason to stop. In the wonton example, neither the input nor the output clearly establishes that eight pieces are present. The system cannot confidently verify the quantity, so it rejects the result in production. The unsure outcome has an operational consequence: a plausible-looking image does not pass when the relevant claim cannot be checked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make publication a separate decision
Passing editing QA is not the final publication decision. The candidate still goes through post-processing and publish-ready QA, including additional policy and quality checks. This last gate takes a more holistic view and can flag issues that earlier stages should have caught.
Chopra explains the redundancy through the Swiss cheese model: each checking layer has gaps, so the system uses overlapping layers to reduce the chance that a failure reaches production. Some checks repeat deliberately. The purpose of the final gate is to reduce escaped failures, even when an upstream evaluator has already approved the candidate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend diagnosis beyond the model evaluation loop
The human-label loop handles drift and alignment with the offline benchmark, but it is only one source of feedback. Internal testing before launch and experience with the live product produce additional signals. To accommodate them, Uber places the diagnoser above the individual agents: it accepts feedback from different loops, identifies which component needs improvement, and routes configuration work to one or several agents.
Internal testing collects thumbs-up and thumbs-down judgments as well as free-form feedback. Merchants, design teams, and other Uber product teams contribute observations that feed back into diagnosis and tuning. Before pushing the latest configuration, the system replays flagged examples—including good examples as well as bad ones—and benchmarks the metrics. A complaint can trigger investigation, but configuration promotion still depends on replay and evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate what happens after deployment
Production evaluation extends to marketplace quality and health. Chopra highlights conversion: whether customers add items to their carts, convert, and complete orders. These are monitored objectives; the talk reports no numerical conversion lift.
At production scale, the data can be sliced by geography, device type, and dish type. Those slices reveal where outcomes improve and where particular segments may need different tuning. The learning process therefore reaches beyond whether an edit passed QA: it follows the deployed system into the marketplace, then uses segment-level evidence to guide subsequent changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Documentation for the observability platform named in the talk, explaining traces, spans and recorded execution metadata.
Further reading
A practical breakdown of evaluating routing decisions, individual skills and agent execution paths.
Official financial results separating Delivery marketplace gross bookings from Uber's recognized revenue.
Read the complete timestamped transcript
- 0:00
[upbeat music] My name is Jai, uh, and I'm here with Soumya.
- 0:19
We are part of the computer vision team at, at Uber, and we're gonna talk to you about a real world production use- Mic's not on. Oh, mic's not on?
- 0:28
Okay. Try again. Okay. Don't worry, I'll, I'll manage. Can you hear me now? Um, okay. So we're gonna talk to you today about a real world production use case.
- 0:41
Um, and specifically we're gonna dive into how we design the evals and the eval loops. Um, so let's see this.
- 0:52
Okay, it's not working. All right, cool. So just before we get into the agent design, um, we're gonna talk about, a little bit about the use case. So our delivery marketplace, Uber Eats, we do about ninety billion, uh, run rate per year at the moment.
- 1:08
Um, we-we're adding millions of items to the marketplace, uh, each and every year. Um, sorry, every, every month. We're growing at twenty percent, uh, year on year, and, and we operate in ten thousand cities globally.
- 1:21
So not many people actually know this, but our delivery marketplace is just as big as the mobility side on Uber today.
- 1:30
Visual content actually plays a really important role for the user experience. So a photo is quite often the first signal that a customer gets, um, that gives them the, that initial impression about a merchant.
- 1:44
Um, so a, a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding to the cart. And more and more we're seeing different modalities, uh, on Uber Eats, uh, especially video content.
- 1:59
But this is a problem. So our smaller independent merchants simply just don't have the level of quality for their photos that reflect what the eater is actually gonna get.
- 2:11
And when we speak to our merchants, there are three themes that kind of emerge: lack of time, lack of know-how, and costs. 'Cause these professional, um, photo shoots actually cost a lot of money, and this can be especially problematic if the merchant is updating their menu over time.
- 2:32
So this problem is actually pretty challenging to solve for at scale, right? Because our consumers, they want authentic, real-looking photos. Um, but a meaningful fraction of our consumers actually distrust anything that is AI-generated.
- 2:48
So if you open up the Uber Eats app, the last thing that you want is to be scrolling through, uh, you know, food photography that looks like AI slop.
- 2:56
So we're threading the needle here. We need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same.
- 3:06
If we have the same prompt for every photo that we're editing, the diversity of the marketplace is gonna collapse.
- 3:16
We also, because we operate globally, we also have this long tail distribution of different quality that we see a-across the marketplace. Um, so we've got some examples here. You might see food photography that, you know, has poor sharpness, poor composition, not centered, uh, or, or poor colors as well.
- 3:35
We also have a wide range of spectrum of user-generated content on the platform as well.
- 3:42
So what are our goals when we're designing these agents? When you think through these goals, you might actually be thinking through, you know, your own agents that you're building yourself.
- 3:50
But for us it's about, one, preserving authenticity and trust. Two, improving the quality when we need to, so we wanna be able to improve qu-quality selectively. We wanna optimize globally for, for the entire marketplace.
- 4:03
We don't wanna canni- cannibalize certain merchants. We wanna ship safely, and this is gonna be an important theme throughout the talk. We want to learn continuously, and we wanna operate at scale in a cost-efficient manner.
- 4:18
So agents are actually really well-suited to solve this problem. So if you imagine a spectrum, on the one side you've got something that's more deterministic, it's more rules-based, um, and, uh, you, you, you have more control over it, but it's fairly, it's a brittle system.
- 4:36
It's not actually gonna be able to scale for the entire marketplace. Imagine the other side. You provide an agent with obviously a lot of creativity, it has a lot of agency, um, and that's actually what we wanna lean into.
- 4:49
But we can't leave that unconstrained, right? Because we have certain safety and certain guardrails in place that we need to adhere to. So we wanna find a balancing act, uh, and that's kind of set the principle for the way that we think and design around agents and evals.
- 5:05
So now we're gonna actually like dive a little bit deeper into a simplified but representative example of what we have in production, and we're gonna go through each stage and how we eval it, and then talk through some continuous learning loops as well.
- 5:21
So first up we have what we call an image understanding and routing agent. So this is where multimodality is, is pretty important. We actually ask the LLM to describe what it sees in the photo.
- 5:33
Um, and then we, we create a structured output from that and we send it to a router. The router will then determine do we enhance it or do we skip it?
- 5:42
If we skip it, we will keep the original. If we enhance it, we send it to our next agent, which is an image editing agent. And this can actually run in a loop, so it gets feedback from a QA agent.
- 5:56
Um, it can edit, uh, uh, uh, in, in this loop and self-correct and fix things, um, as it goes.
- 6:04
If it goes through a number of loops and it still fails, we, we don't publish it. Then we actually send it to a final post-processing and QA step. If that's all good, we'll publish it to the menu.
- 6:17
And the last thing that's really critical is we log everything.
- 6:23
Just a quick note about logging. Don't know if you can actually read the JSON here, but you might notice that all of the agents in this end-to-end orchestration is within one-- it's, it's basically a flat structure in this JSON.
- 6:37
Um, and so this is actually incredibly useful for the entire team because anyone, be it non-technical or technical, um, focused on engineering product, can actually dive in, um, and look at specific cases to diagnose and also roll up things to look in aggregate.
- 6:54
Um, and it's important to note here that, you know, we think this is important to start with. You wanna start with your logging, 'cause if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.
- 7:07
And at Uber, we, um, we use Arize.
- 7:11
Cool. We're gonna dive, um, a bit deeper into the router.
- 7:17
So the router's actually pretty straightforward. If you remember, we, you know, we have this multimodality input. We look at certain text description, metadata, the image itself. We ask it to, to, to, to under-- um, describe what it's seeing.
- 7:29
We create structured output from that. With that structured output, we can then grade against a rubric. So we have these pass and fail criteria. The last step is we wanna decide whether or not we should enhance or skip.
- 7:43
How do we actually eval this? This is-- You can think of this as a more sort of traditional classifier. So here we, we have a confusion matrix. You know, many of you are probably pretty familiar with this.
- 7:54
Um, but we can look at things like the true positive cases, the false ne-negative cases, and so on and so forth. Essentially, what we're doing is we're measuring the precision recall.
- 8:03
In practice, your routers might actually be much more sophisticated. So for example, we might wanna route an image to a lower latency, smaller model to be able to save on cost and improve the user experience at the trade-off of quality.
- 8:17
And i-if that's the case, instead of having a two-by-two matrix for your confusion matrix, you might actually have an end-by-end matrix, where each grid is actually telling you whether or not you're correctly routing to that specific branch.
- 8:32
So I'm gonna now, um, hand over to Soumya, who's gonna dive a little bit deeper into how we handle drift and human alignment.
- 8:41
Do I-- Do I take the clicker as well?
- 8:45
Okay.
- 8:48
So now that we spoke about how we eval the routing, I wanna talk about how do you get the first version of the model out. For our use case, we consider human labels as the golden source of truth, and this is what we want to align our models to.
- 9:01
The way we do about this is we go collect a dataset which is representative, so, you know, different cuts, geographies, dish type, image quality type, send it to our human labelers, and give them a very objective guideline to label on.
- 9:14
This is to remove any subjective biases or any noise coming in from human labelers. Once we've got that system set up is when we start tuning our model. We take our agent, we go ahead and get output from the agent, compare it to your golden dataset, evaluate if it's good enough to ship.
- 9:28
If it meets your guardrail metrics, you go ahead and ship it. If not, then you go tune and keep doing this until you meet your guardrail metrics. For routing, our guardrail metric is recall.
- 9:38
We don't want any bad image to slip through our system.
- 9:44
Here are some examples of the failures we've seen. Um, on your left, you see a very good image of cheeseburger. Um, on the right, you notice that the routing agent actually failed this.
- 9:53
It said the technical is low bal, and it'll go send this image for enhancement. Now, there's two challenges when you send this image for enhancement. Firstly, you pay the compute cost for a zero quality lift from this image.
- 10:04
And secondly, uh, there is a risk of degrading this image, given it's already such a high-quality image.
- 10:13
And on the other end of the spectrum, you have a recall miss. So on your left, you have an image with six chicken wings, and on your right, if you notice the dish name, it says eight pieces chicken wings.
- 10:23
And your routing agent approved this image, that means. So now there's a risk here. If you send up-- send this image for enhancement, and you only see six chicken wings, there's a chance your model is gonna hallucinate these two extra wings just to match the description.
- 10:36
And that's also an, uh-- That's a, the co-- um, cut we take at our faithfulness metric that Jai earlier showed us. So the meta point I'm trying to get here is you've trained your offline model, but there will be long cases where your model is going to continue to fail, and the static model will not work in
- 10:51
the real system. You need a way such that your prompts, agents, system itself is evolving over time. And that's what we've done, uh, for our system as well. And I'm talking more from the routing perspective, but every component in our system is able to tune itself, uh, for any drift online.
- 11:08
So what we do is we sample production data at regular cadence, uh, send this to the human labelers with the same guidelines that we have se-seen before. Once you've got that data, we compare our agent's output with the output we got from the labelers and see if there's a mismatch.
- 11:22
If there's a mismatch, we have an umbrella diagnoser agent, which takes in the feedback, localizes where this issue is happening, and sends-- and triggers our autotuning pipeline. Once we tune this agent, we go and benchmark it against our golden dataset that we saw earlier.
- 11:38
And if we pass our golden dataset on the metrics that we had designed, we go ahead and ship this model. Uh, if not, then you kinda keep iterating. And this happens on a regular basis on production dataset.
- 11:49
Um, the beauty of this is this is completely config-driven and doesn't require human-in-the-loop. Your diagnoser agent can write your config and trigger the autotuning pipeline here. And this is what will keep your models sharp over time.
- 12:01
You will have one static model with the offline, but this is what is going to keep your system alive.
- 12:08
Um, so Jai is gonna spend more time on the diagnosis side of it. What I want to do is zoom into the auto-tuning bit. And again, we are looking at routing, but this is how we tune every agent in our system.
- 12:20
Um, so we start with a target agent, and we've already got these, uh, unseen eval samples from our humans. We go find out the mismatch and matches and call a prompt optimizer agent.
- 12:30
Now, this itself is two sub-agents. There's the ref-reflect agent and the op-- synthesize agent. What reflect does is it, it just looks at the mismatches, tries to find... remove any noise, find any systemic issues that might be in your dataset, and reflect on it and send that feedback to the synthesize agent.
- 12:48
Now, the syn-synthesize agent takes this feedback. It has your agent config. It goes and updates your agent with the new config based on the feedback it's getting and goes and benchmarks again.
- 12:57
If this benchmark is passed, you actually register this new agent in the new agent config store, and next time your production runs, you pick up the new version of the agent.
- 13:07
And this is a c-closed loop system, as I mentioned, no human in the loop. We definitely have observability on the guardrails. Quick rollback built in in case of, uh, any issues with the system itself.
- 13:20
Moving on to the next step of our orchestration flow. So we spoke about routing, moving on to the enhancement bit of it. It's a three-step process. What we do is the first step, we generate a prompt specific to this image.
- 13:32
We take in the this description, we take in the directives we were getting from our routing agent, and we go ahead and generate a prompt for this image, what needs improvement in this image specifically.
- 13:40
And we go ahead and enhance this image. Then you've got the QA gate, which is a multidimensional gate, looks at multiple things like plating, faithfulness, colors. And it, if it passes is when you actually go ahead and publish this.
- 13:52
If it doesn't pass, you take the feedback back from the QA gate, push it back to your generate prompt along with the initial inputs you sent it, and go ahead and enhance it again.
- 14:00
So there's two end results here. You either keep enhancing for K iterations, and you pass your QA gate, and you publish, or you take a coverage hit, and you never enhance this image.
- 14:11
Here's an example. On your left, you see a bowl of sweet potato fries. We send it, uh, for the first iteration, and our QA agent rejects it because the portion size is in-incorrect.
- 14:20
The plating is very unrealistic. We take that feedback in, go for the second iteration, and we're actually able to pass it the second iteration. So the metric we are measuring here is pass at K.
- 14:30
Pass at K is essentially the pass rate at K at iteration. And ideally, with the more the iterations, your, uh, pass rate will increase because you're getting more feedback in.
- 14:40
Now I'll pass it on back to Jai to cover the rest of this.
- 14:47
Thanks. Thanks, Soumya. Um, so yeah, just before we end here on the, um, on, on the generation evals, we use what's called pairwise comparison, right, for our pass at K.
- 15:01
So it's looking at the input image and the, the output image, and it's assessing whether or not it's better. But how do we actually find what's better? So, um, we're not gonna dive into too much of the details here 'cause this is kind of like proprietary stuff.
- 15:16
And so we'll just mention at a high level that this is where you sort of, for at least for us at Uber, we have to make sure that we're aligning with product, design, policy, legal.
- 15:26
And this is where we're baking in what we define as a better image on the platform into our evals. Um, so examples here, is it faithful? Is it complete?
- 15:36
Is it natural? Is it realistic? And there's a bunch of other things as well. The output of this is then, uh, a yes, no, or unsure.
- 15:45
So here are some examples of failure modes. So input and output on the right. The input's on the left, output's on the right-hand side. This might be a little bit, uh, difficult to, to see at a first pass, but we actually added shrimp here, and we shouldn't be.
- 15:59
So we fail faithfulness. This is where we go the other way. So the input, um, has some sauce at the bottom of the sushi. We actually remove it. So we fail completeness.
- 16:15
Here's actually a, a pretty interesting example where the agent actually attempted a more creative edit the first iteration, um, and then the QA said, "Nope, that's not good enough."
- 16:26
Uh, and then it actually oversteers the other, other way, and it becomes overly conservative, sort of falls back to this generic ceramic plate, uh, ceramic bowl, sorry. So this is an example of a reward hacking actually.
- 16:40
And, and this is a nugatory change, but it's something that we don't think is a meaningful or influential change despite the actual raw pixels of the input and output being pretty different.
- 16:50
Here's another example where in the output, the plate is covering the sauce. This is an example where for... So some of the frontier models that we're using for the actual image editing, some of their, um, some of their, um, problems will actually sort of leak up into our applied use case.
- 17:06
Um, and so, so object coherence and ph-physics plausibility are the evals that sometimes we'll coordinate with the frontier teams and, and let them know about these problems and work together with them.
- 17:18
Here's, uh, an example of why multimodality is, is pretty important. In the input and the output, we, we can't actually see that there are eight pieces here of, of the wontons.
- 17:28
So we're not confident actually. We're not sure. And so this is an example where we would actually reject it in production and, and it wouldn't, it wouldn't go through.
- 17:39
So the last step after all of that is a post-processing and what we refer to as the publish-ready QA. This is the final gate before we decide we wanna publish something to production.
- 17:53
Here we do some policy checks. We also do some more quality checks. Um, and you might be wondering like, "We've already done some QA, like, why are we gonna do another step of QA?"
- 18:04
The reason is because we think of this like a Swiss cheese model. So we want to try and optimize for reducing the chance of a failure getting into production.
- 18:15
And so there is some redundancy here or there, and that's okay. Um, and so this QA gate is, is a little bit more holistic. It captures more things, but it also will, will try and flag things that we should have caught upstream as well.
- 18:33
All right, so we've talked about a, a couple of, uh, feedback loops here. So to summarize, we talked about predominantly this first one here, which is the model loop.
- 18:44
And this is accounting for drifts and aligning with human label dataset that we have and we've established offline. But we actually have more feedback loops. So we, we have at Uber, what we have is a, is a great sort of dog, dog fooding culture, um, where we'll test apps before they go live.
- 19:03
Um, but we also have when it goes live in production, how do we get that feedback back into our agent to be able to steer it appropriately? So as we're adding more of these feedback loops, we want to be able to generalize the system.
- 19:17
So this is where we've actually created, um, a higher level of abstraction on top, which we call the diagnoser. So the diagnoser can take in any input from these different feedback loops that we're, we're capturing.
- 19:29
It can reflect on what actual agent within the overall system needs to be optimized, and it can route to that agent to be able to fix that configuration specifically.
- 19:40
It could be one agent, it could be multiple agents.
- 19:45
So here's an example of internal dog fooding. You might see these in sort of different apps that you've got where you've got the thumbs down and the thumbs up.
- 19:52
We also take some free form feedback as well. Uh, and this is actually great 'cause we'll get feedback from merchants directly. We'll get feedback from, you know, design teams, other product teams, uh, at Uber, and we'll incorporate that feedback back into our diagnoser step and tune the system over time.
- 20:11
Again, similar sort of workflow h- pattern here. We'll replay the examples that we know are those ones that have been flagged, be it good examples, be it bad examples, uh, and then we'll benchmark the metrics before we push the latest config version.
- 20:27
The last step is, is actually getting this into production and, and this is where we're looking for a whole heap of different metrics that we track for, for the marketplace quality and health.
- 20:38
Uh, I've just called out one here, which is conversion. So we're looking for improvements in people adding to cart, converting, completing their orders. Um, I think this one's actually an interesting one to call out because now, I mean, at least at Uber, but especially in production, um, settings at scale, you have a wide, um, uh, you, you
- 20:57
have a lot of data that you can actually slice and dice. So in this area as opposed to the others, what we can do is sort of slice by geos, by device type, by dish type, et cetera.
- 21:08
And we can look at where things are improving, uh, in different segments and actually tune on certain segments as well.
- 21:17
Cool. And that's it for our presentation. Appreciate it. [audience applauding] [outro music]