Building Closed-Loop Evals for a Multimodal Agent at Uber Scale

How can a delivery marketplace improve millions of food photos while keeping them representative of what customers will receive? The scale makes this an engineering problem as well as a photography problem. Jai Chopra reports an approximately $90 billion annual marketplace run rate, 20% year-on-year growth, operations in 10,000 cities, and millions of items added each month. These are the talk’s contextual figures, with no reference period specified; the marketplace run rate should not be read as Uber revenue. Chopra describes the delivery marketplace as comparable in size to Uber’s mobility business.

A photo is often a customer’s first impression of a merchant. It can determine whether someone keeps scrolling, opens an item, or adds it to the cart. Video is becoming more prominent too, expanding the visual content the marketplace must handle. But independent merchants often have photos that do not reflect the quality of their food. Their constraints are practical: time, photography know-how, and cost. Professional shoots are expensive, particularly when changing menus require another round of photography.

Better-looking images are useful only if customers trust them. Chopra describes consumer distrust of AI-generated imagery as a constraint on the product: an edit must remain faithful to the original food, preserve the merchant’s brand, and avoid making every restaurant look alike. A single editing prompt can collapse marketplace diversity, even if its individual outputs look polished.

The inputs also resist a uniform treatment. Across a global marketplace, photographs have different combinations of poor sharpness, off-center composition, weak colors, and other quality problems. User-generated content adds further variation. Enhancement therefore needs to be selective: improve what needs improvement while preserving what already works. The goals extend beyond individual images to marketplace-wide outcomes, avoiding gains that simply cannibalize other merchants, shipping safely, learning continuously, and keeping operating costs manageable.

There are two unhelpful extremes. A deterministic rules system provides control but becomes brittle across the long tail of images. An unconstrained agent has the creative freedom to generalize, but can violate the safety requirements that make an edit acceptable. Uber’s design gives agents room to act inside explicit guardrails. The architecture that follows is a simplified but representative account of the production system.

The execution path begins with image understanding and routing. A multimodal LLM describes the photograph and produces structured output. A router uses that assessment to choose between enhancement and skipping. Skipping keeps the original image. Enhancement sends the image to an editing agent, which receives feedback from a QA agent and can revise its work in a loop. If repeated attempts still fail, the system does not publish the edited image. A successful candidate proceeds to final post-processing and QA before menu publication.

Every stage is logged. In the representation Chopra describes, the agents’ records sit together in a flat JSON structure, making a complete case accessible to both engineering and product colleagues. The same records support aggregate analysis. An illustrative record for the skip path makes the decision and the absence of downstream work explicit:

{
  "image_understanding": {
    "assessment": "No enhancement needed"
  },
  "router": {
    "decision": "skip"
  },
  "image_editing": null,
  "editing_qa": null,
  "publish_ready_qa": null,
  "outcome": "keep_original"
}

The field names illustrate the flat arrangement rather than reproduce Uber’s schema. Its value is that someone investigating a case can see both what happened and which stages were never invoked.

Logging precedes optimization. Without execution records, there is little evidence with which to diagnose failures or build a learning loop. Chopra names Arize as Uber’s observability platform. The flat JSON arrangement described here is Uber’s execution representation, not a claim that Arize’s general trace model is flat.

The router combines the image with its text description and metadata. The model describes what it sees in structured form; that output is graded against a rubric with pass/fail criteria, producing the enhance-or-skip decision. This separates the visual assessment from the routing choice and gives the team a classification problem to evaluate. A confusion matrix exposes correct decisions and misses, with precision and recall measuring different aspects of the result.

More elaborate routing can choose an execution strategy as well as decide whether to edit.

For example, a branch might send an image to a smaller, lower-latency model to reduce cost and waiting time at a possible quality cost. Each cell in the larger matrix then helps expose whether inputs reach the appropriate branch.

For the first release, Soumya Gupta treats human labels as the reference the model should match. The process starts with a representative dataset spanning geography, dish type, and image quality. Labelers receive objective guidelines intended to reduce subjective bias and noise. The team then runs the agent, compares its outputs with that golden dataset, and tunes until the release guardrails pass. For routing, the guardrail is recall: the system should not let bad images slip through.

The cheeseburger example shows the cost of unnecessary intervention. A high-quality photograph is incorrectly sent for enhancement. Gupta describes this as paying compute cost for no quality lift, while also exposing an already good photograph to degradation. Correctly deciding to leave an image alone is part of the product’s quality control.

The chicken-wing example exposes a different risk: the photograph shows six wings, while the dish description specifies eight pieces. Gupta presents the router’s approval as a recall miss. The downstream hazard is an editor trying to reconcile that mismatch by generating two additional wings. Those additions would match the text while violating faithfulness to the photograph. The example makes multimodal evaluation consequential: an image and its description can disagree, and an edit must not silently resolve that disagreement by inventing food. These long-tail failures remain possible after the offline model has passed its initial evaluation.

A static model cannot cover a changing production distribution indefinitely. Uber regularly samples production data and sends it to human labelers using the same guidelines as the offline benchmark. Agent-label mismatches go to an umbrella diagnoser agent, which localizes the problem and triggers autotuning. The tuned agent must pass the established golden-dataset metrics before it can ship; otherwise, the loop continues. Although routing is the example, Gupta describes this capability as applying to every component.

The diagnoser can write configuration and trigger the tuning pipeline without a person manually orchestrating those steps. That automation still depends on the human-labeled samples entering the loop. The distinction matters: human judgments establish the target, while configuration-driven machinery carries out diagnosis, optimization, and release evaluation.

Inside autotuning, the prompt optimizer divides its work between Reflect and Synthesize:

1. Evaluate the target agent on unseen human-labeled samples, identifying matches and mismatches.
2. Have Reflect examine the mismatches, filter noise, and identify systematic problems.
3. Pass that feedback and the current agent configuration to Synthesize, which produces an updated configuration.
4. Benchmark the updated agent. If it passes, register the new version in the agent configuration store.
5. Let the next production run pick up the registered version.

This makes a proposed configuration distinct from a configuration eligible for production: the benchmark sits between them.

The automated loop also has observability on its guardrails and a quick rollback mechanism. Benchmarking controls promotion; monitoring and rollback address problems that become visible after promotion.

Once an image reaches enhancement, the system generates a prompt specifically for it. Prompt generation takes the description and the routing agent’s directives about what needs improvement. The resulting prompt drives image enhancement, followed by a multidimensional QA gate that examines properties such as plating, faithfulness, and colors.

A failed candidate returns QA feedback to prompt generation alongside the original inputs. The next attempt therefore has both the initial task and evidence about what went wrong. This is a bounded loop: within K iterations, the system either obtains a candidate that passes the editing gate or accepts a coverage hit and produces no accepted enhancement. It does not have to publish an edit merely because it spent compute generating one.

In the sweet potato fries example, the first edit fails QA for incorrect portion size and unrealistic plating; the second edit passes after receiving that feedback. Gupta calls the metric pass at K, describing it as the pass rate at the Kth iteration and expecting additional feedback to improve acceptance. The talk does not specify whether the aggregate rate is cumulative through K attempts or measured only at iteration K, so that definition must be made explicit before implementing the metric.

Generation evaluation uses pairwise comparison: assess the input image beside the output and determine whether the output is better. That requires a product definition of better, not just an aesthetic preference. Uber aligns the rubric with product, design, policy, and legal. The detailed criteria are proprietary, but the disclosed dimensions include faithfulness, completeness, naturalness, and realism. The resulting judgment can be yes, no, or unsure.

Two examples separate adding unsupported content from losing existing content:

Both outputs may appear visually plausible. Comparing them with the original reveals why neither is an acceptable enhancement.

Feedback can also steer an editor toward an unhelpful solution. In one example, a creative first edit is rejected, and the next attempt overcorrects toward a generic ceramic bowl. Chopra describes this as reward hacking: the agent becomes overly conservative in response to QA. The pixels can change substantially without producing a meaningful improvement. An evaluator that rewards change alone would miss the distinction between a different image and a useful edit.

Another output has a plate covering the sauce. This is a failure of object coherence and physical plausibility, and it illustrates how limitations in the underlying image-editing models reach the applied product. Uber sometimes shares these failures with frontier model teams and works with them on the underlying problems. Application-level QA thus identifies both mistakes the agent configuration might address and weaknesses in the model doing the editing.

Uncertainty is itself a reason to stop. In the wonton example, neither the input nor the output clearly establishes that eight pieces are present. The system cannot confidently verify the quantity, so it rejects the result in production. The unsure outcome has an operational consequence: a plausible-looking image does not pass when the relevant claim cannot be checked.

Passing editing QA is not the final publication decision. The candidate still goes through post-processing and publish-ready QA, including additional policy and quality checks. This last gate takes a more holistic view and can flag issues that earlier stages should have caught.

Chopra explains the redundancy through the Swiss cheese model: each checking layer has gaps, so the system uses overlapping layers to reduce the chance that a failure reaches production. Some checks repeat deliberately. The purpose of the final gate is to reduce escaped failures, even when an upstream evaluator has already approved the candidate.

The human-label loop handles drift and alignment with the offline benchmark, but it is only one source of feedback. Internal testing before launch and experience with the live product produce additional signals. To accommodate them, Uber places the diagnoser above the individual agents: it accepts feedback from different loops, identifies which component needs improvement, and routes configuration work to one or several agents.

Internal testing collects thumbs-up and thumbs-down judgments as well as free-form feedback. Merchants, design teams, and other Uber product teams contribute observations that feed back into diagnosis and tuning. Before pushing the latest configuration, the system replays flagged examples—including good examples as well as bad ones—and benchmarks the metrics. A complaint can trigger investigation, but configuration promotion still depends on replay and evaluation.

Production evaluation extends to marketplace quality and health. Chopra highlights conversion: whether customers add items to their carts, convert, and complete orders. These are monitored objectives; the talk reports no numerical conversion lift.

At production scale, the data can be sliced by geography, device type, and dish type. Those slices reveal where outcomes improve and where particular segments may need different tuning. The learning process therefore reaches beyond whether an edit passed QA: it follows the deployed system into the marketplace, then uses segment-level evidence to guide subsequent changes.