Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

Nidhi Kaushik Vyas, a product person at Google DeepMind, introduces collaborative agents through shopping tasks where intent is still fuzzy. The user may have a desired feeling without the keywords needed to describe it. In this setting, guiding the user requires proactive preference elicitation as well as execution. Commerce makes these patterns easy to illustrate, though she also suggests their relevance to other consumer domains, including finance and education.

Her diagnosis is that many agents behave like a wrapper around a search bar. They assume the shopper already knows what they want and can express it in the system’s vocabulary. The articulation gap appears before retrieval: users often arrive with an incomplete preference or a vibe, so answering their initial query alone may leave the underlying goal unresolved.

The proposed agent works with the user to uncover preferences, show possible directions, and eventually recommend options that satisfy the resulting constraints. Showing possibilities matters because the shopper may recognize a preference before being able to name it. Vyas frames this progression as a loop from fuzzy intent toward achieving the user’s goal.

Discovery gathers the context already available: past conversations, the current query, personal context, and references supplied by the user. It then develops a collaborative strategy for obtaining whatever additional information would clarify the intent. This separates remembering what is known from deciding what still needs to be asked.

Research has two responsibilities. First, it chooses how to elicit a preference. A text question can fail when the user lacks the language to answer it, whereas visual references or inspiration boards can establish a shared vocabulary. Second, the agent performs comparisons, examines tradeoffs, summarizes information, and returns potential options. The intended benefit is to reduce the descriptive and analytical burden placed on the shopper.

Response adapts the presentation to the task. A useful result might take the form of a bulleted list, a comparison table, or a visual board. Vyas treats this choice as part of the agent’s competence: the system must make the answer easy to find and use, rather than defaulting to a text-heavy response regardless of the query.

The detailed discovery example concerns a user who wants to redo a living room within a budget. The agent assembles a working state from session history, user context, and explicit hard constraints in the query. Past conversations, reference images or links, and personal context all contribute to this representation. The budget is described as a constraint, but no amount is specified in the spoken example.

Soft constraints require inference. A reference image might express an appealing style or layout that the user cannot describe directly. The agent extracts salient signals from that image and develops a tentative model of the preference. It also attaches confidence to the extracted information. This distinction matters: an image-derived style hypothesis carries uncertainty that an explicitly stated constraint may not.

The state must also identify variables that need near-real-time refresh. Inventory is the concrete example: a recommendation based on stale availability may be unusable even if it otherwise matches the shopper’s preferences. Freshness therefore becomes part of constructing a useful working state, because changing operational facts affect which results can be shown.

Automatic evaluators check whether the working state retains the facts supplied in context and whether its confidence estimates stay within an error bound. Vyas does not give a numerical bound or a scoring formula. She also describes counterfactual sensitivity checks: change part of a query and verify that the affected extracted constraints change, while unrelated constraints remain stable. This tests both responsiveness to relevant evidence and resistance to unnecessary changes elsewhere in the state.

Once the agent represents what it knows, discovery turns to the intent gap: what remains unknown before a useful recommendation can be made? In the living-room example, missing information includes room width and stronger evidence about style preference. Identifying these unknowns does not mean asking about all of them upfront. The agent needs a strategy for choosing among possible next moves.

Vyas proposes comparing possible moves and prioritizing the unknown with maximal information gain at that point in the conversation. Room width is a plausible first choice because furniture that cannot fit is not a meaningful recommendation. Its answer can substantially change the direction of the search and conversation. The example explains the decision principle through feasibility; it does not specify a mathematical estimator for information gain or establish that room width is always the best question.

Evaluation must balance uncovering blockers with avoiding excessive questioning. The agent should identify the information required for a meaningful answer, but repeated clarification can trap the user in a loop. Evaluators flag over-asking and assess question utility: whether a question actually elicits the preference needed to proceed. The practical tradeoff is between gathering more information and making useful progress with each interaction.

The example advances after the user supplies room width. The next constraint to explore is style. Before retrieving products, the agent forms a temporary bridge between the user’s constraint and the product catalog’s ontology—the organization of product knowledge and metadata. This mapping connects the preference being explored to information the knowledge base can use. Vyas describes the bridge as being formed almost in real time, without detailing its schema or retrieval algorithm.

The agent also chooses the form of elicitation. Because style is subjective, it may use a visual preference board instead of asking the user to name a style in text. The board’s options draw on existing constraints, previous conversation, and the temporary mapping to product knowledge. In this example, styles similar to the supplied reference image provide starting points the shopper can recognize and compare.

Interaction with those options supplies additional evidence. The agent can observe micro-signals such as hovers and clicks and use them to update confidence about the shopper’s style preference. These signals contribute to a confidence model; the presentation does not provide weights or a rule that turns a particular click into a confirmed preference.

To evaluate preference discovery, Vyas describes giving a user simulator known constraints and measuring how efficiently the agent elicits them. Turn efficiency checks how many interactions are needed and whether questions repeat or fail to provide useful information. Format selection is evaluated separately: an easily stated answer can suit a text question, while a preference the user struggles to articulate may need visual anchors. These are described evaluation methods, without reported scores or evidence that simulator performance alone establishes performance with real shoppers.

The final phase assumes the conversation has clarified the relevant preferences and priorities. For the living-room task, the agent now has a style preference, a budget constraint, and dimensions that guide product selection. The conversation also helps identify which product metadata should be surfaced. The response must therefore reflect what matters to this shopper, including the details needed to judge the options.

Vyas gives three examples of task-dependent presentation. A question about product policy or reviews may call for a summary or bulleted list. Comparing two products may call for a table that exposes tradeoffs along the axes the user cares about. Style inspiration for a room may call for visual references and product inspiration photos that reveal possible directions. Each format serves a different decision: understanding information, comparing alternatives, or exploring an appearance.

Response evaluation checks format accuracy, data fidelity, and user actionability. The needed information should be easy to spot, and product facts should remain accurate throughout the response. Actionability asks whether the presentation helps the user feel confident enough to take the next step, which Vyas identifies here as purchasing. These criteria distinguish an attractive response from one that preserves the data and supports a decision; the talk does not report purchase outcomes or quantitative improvements.

Vyas closes the presentation with four principles. Accept fuzzy intent and queries that arrive without a well-defined goal. Show options as well as asking questions, using visuals and comparisons to establish a common language. Shape the answer so the user can find the relevant information. Grade the entire loop. Her central product claim is that response structure belongs within the agent’s intelligence, alongside its ability to understand and reason.

The evaluators themselves are an evolving system. They can start simply, then grow as the agent develops, with automatic evaluation at every step of the process. This makes evaluation an ongoing part of building the product rather than a fixed check applied only to its final response. Vyas ends the prepared talk by inviting questions and suggesting that these lessons can inform what the audience is building.

In the questions, Vyas addresses how a merchant-side ontology should support both merchants and agents. Merchant domain expertise helps connect user constraints to the merchant’s product metadata. She places this expertise in the bridge between what the shopper expresses and what the agent can understand through the catalog. She also mentions UCP as helping merchants speak a common language with the agent, but does not describe its precise scope or implementation, nor give a detailed fairness mechanism.

She then clarifies who controls presentation. Information flows back to the agent, which makes decisions to provide a common layer across merchants and a seamless user experience. Response format is part of the agent’s intelligence rather than something each merchant decides. This assigns merchants an important role in supplying domain knowledge while keeping the user-facing presentation under the agent’s control.

An audience member asks what happens when the user is another agent. Vyas describes this as early-stage work and says she would expect MCP to serve as an interface between the two. She explicitly says they have not yet reached the point of agents interacting with their agents. The interface is therefore an expectation, with no demonstrated agent-to-agent workflow or implementation detail supplied.

Her reported user studies suggest a boundary for delegation. During discovery and inspiration, users like being involved in choosing and exploring possibilities, and would rather interact directly with the system. An agent may be more useful later, when the task becomes comparing options, negotiating, or comparing prices across merchants. This qualifies the push toward greater agency: shoppers may want assistance in forming their preferences while retaining participation in that process. Vyas closes by offering to continue questions outside and thanking the audience.