[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13880,
    "text": ">> Okay, good morning folks. Thank you for"
  },
  {
    "startMs": 16080,
    "text": "showing up. I'm Nidhi. I'm a product"
  },
  {
    "startMs": 18520,
    "text": "person at Google DeepMind. And today"
  },
  {
    "startMs": 20720,
    "text": "I'll be talking about multimodal"
  },
  {
    "startMs": 22760,
    "text": "collaborative agents. Typically, these"
  },
  {
    "startMs": 24720,
    "text": "are agents that work with fuzzy intent"
  },
  {
    "startMs": 27520,
    "text": "intent when the intent when it's not"
  },
  {
    "startMs": 29560,
    "text": "clear the user hasn't gotten the right"
  },
  {
    "startMs": 31520,
    "text": "keywords to specify and when they come"
  },
  {
    "startMs": 33640,
    "text": "in for a shopping intent, how can you"
  },
  {
    "startMs": 35560,
    "text": "have the agents still guide the user"
  },
  {
    "startMs": 38200,
    "text": "towards their goal with high agency"
  },
  {
    "startMs": 40120,
    "text": "execution, proactive elicitation, and a"
  },
  {
    "startMs": 42720,
    "text": "lot of hand-holding."
  },
  {
    "startMs": 44160,
    "text": "The framework that we'll be discussing"
  },
  {
    "startMs": 45800,
    "text": "today are grounded in shopping or"
  },
  {
    "startMs": 48080,
    "text": "commerce because I wanted to show you a"
  },
  {
    "startMs": 50240,
    "text": "few patterns that are much easier to see"
  },
  {
    "startMs": 51760,
    "text": "in commerce, but they're quite"
  },
  {
    "startMs": 53440,
    "text": "applicable in other consumer verticals"
  },
  {
    "startMs": 55240,
    "text": "as well, finance, education, or whatever"
  },
  {
    "startMs": 57720,
    "text": "you guys work on."
  },
  {
    "startMs": 59200,
    "text": "Um"
  },
  {
    "startMs": 60280,
    "text": "Yeah, so before we even start into the"
  },
  {
    "startMs": 62240,
    "text": "frameworks, I wanted to discuss about"
  },
  {
    "startMs": 64160,
    "text": "why this is an existing problem."
  },
  {
    "startMs": 65840,
    "text": "Currently, a lot of the agents that we"
  },
  {
    "startMs": 68360,
    "text": "have act more like a wrapper to the"
  },
  {
    "startMs": 70560,
    "text": "search bar."
  },
  {
    "startMs": 72080,
    "text": "They assume that the user has a"
  },
  {
    "startMs": 74360,
    "text": "well-defined intent, has the right"
  },
  {
    "startMs": 76360,
    "text": "keywords, already knows what they're"
  },
  {
    "startMs": 78680,
    "text": "looking for. And so when they come in,"
  },
  {
    "startMs": 81400,
    "text": "they just have the right vocabulary to"
  },
  {
    "startMs": 83040,
    "text": "interact with the agent. However, there"
  },
  {
    "startMs": 85240,
    "text": "is quite a huge articulation gap. When"
  },
  {
    "startMs": 88160,
    "text": "the users come in, rarely they have"
  },
  {
    "startMs": 90080,
    "text": "their intent well formed. Rather, they"
  },
  {
    "startMs": 92240,
    "text": "kind of have a fuzzy feeling or a vibe"
  },
  {
    "startMs": 95360,
    "text": "when they are kind of looking to shop."
  },
  {
    "startMs": 97760,
    "text": "So,"
  },
  {
    "startMs": 99040,
    "text": "the agent needs to play quite a huge"
  },
  {
    "startMs": 101480,
    "text": "role in hand-holding them. They need to"
  },
  {
    "startMs": 104120,
    "text": "work with the user in first"
  },
  {
    "startMs": 105640,
    "text": "understanding their preferences or even"
  },
  {
    "startMs": 107200,
    "text": "eliciting these preferences more"
  },
  {
    "startMs": 108800,
    "text": "proactively."
  },
  {
    "startMs": 110200,
    "text": "And then helping the user"
  },
  {
    "startMs": 112720,
    "text": "kind of show them different"
  },
  {
    "startMs": 113960,
    "text": "possibilities of what could be possible"
  },
  {
    "startMs": 115880,
    "text": "that they can shop"
  },
  {
    "startMs": 117200,
    "text": "and then eventually move towards"
  },
  {
    "startMs": 119520,
    "text": "kind of recommendations that work with"
  },
  {
    "startMs": 121280,
    "text": "these constraints that they have in"
  },
  {
    "startMs": 122520,
    "text": "mind. So, what we're going to be"
  },
  {
    "startMs": 124440,
    "text": "discussing is this kind of a flywheel or"
  },
  {
    "startMs": 126800,
    "text": "a loop that goes from very, very fuzzy"
  },
  {
    "startMs": 130320,
    "text": "intent all the way towards kind of"
  },
  {
    "startMs": 132680,
    "text": "achieving the user's goal."
  },
  {
    "startMs": 135959,
    "text": "So, this is how the shopping loop looks"
  },
  {
    "startMs": 138000,
    "text": "like right now. The very first thing"
  },
  {
    "startMs": 139800,
    "text": "that the agent prepares for when the"
  },
  {
    "startMs": 141520,
    "text": "user comes in is what we call the"
  },
  {
    "startMs": 143720,
    "text": "discovery phase."
  },
  {
    "startMs": 145240,
    "text": "This is where the where the agent takes"
  },
  {
    "startMs": 147320,
    "text": "all the different contextual information"
  },
  {
    "startMs": 149600,
    "text": "that they have about the user. This"
  },
  {
    "startMs": 151160,
    "text": "could be past conversations, what the"
  },
  {
    "startMs": 153760,
    "text": "user has specified in their query. It"
  },
  {
    "startMs": 156040,
    "text": "could be information present in their"
  },
  {
    "startMs": 157800,
    "text": "personal context, or even the references"
  },
  {
    "startMs": 159920,
    "text": "that the user have provided. And then"
  },
  {
    "startMs": 162000,
    "text": "comes up with a collaborative strategy,"
  },
  {
    "startMs": 163720,
    "text": "and we'll be going to the details of"
  },
  {
    "startMs": 164959,
    "text": "this, but it comes up with a"
  },
  {
    "startMs": 165760,
    "text": "collaborative strategy on what more does"
  },
  {
    "startMs": 168440,
    "text": "the agent need to elicitate back from"
  },
  {
    "startMs": 170840,
    "text": "the user in order to help get the intent"
  },
  {
    "startMs": 173640,
    "text": "in a better shape, to get clarity on"
  },
  {
    "startMs": 175280,
    "text": "what exactly the user is looking for."
  },
  {
    "startMs": 177560,
    "text": "And then it moves to the second phase,"
  },
  {
    "startMs": 179000,
    "text": "which is what we call the research"
  },
  {
    "startMs": 180560,
    "text": "phase."
  },
  {
    "startMs": 181480,
    "text": "Which is again a two-step process. First"
  },
  {
    "startMs": 183840,
    "text": "is where"
  },
  {
    "startMs": 185120,
    "text": "it learns what is the best way to"
  },
  {
    "startMs": 187400,
    "text": "elicitate this preference. So, for"
  },
  {
    "startMs": 188959,
    "text": "example, a lot of times the text the"
  },
  {
    "startMs": 191280,
    "text": "text base elicitation may not be the"
  },
  {
    "startMs": 193200,
    "text": "best way to get the preferences from the"
  },
  {
    "startMs": 195000,
    "text": "user because sometimes a user don't know"
  },
  {
    "startMs": 197200,
    "text": "what they're looking for. So, then how"
  },
  {
    "startMs": 198840,
    "text": "can the agent start being more creative"
  },
  {
    "startMs": 200560,
    "text": "in terms of elicitating these"
  },
  {
    "startMs": 202160,
    "text": "preferences? Can they start using some"
  },
  {
    "startMs": 204400,
    "text": "kind of visual references or visual"
  },
  {
    "startMs": 206280,
    "text": "visual inspiration boards for grounding"
  },
  {
    "startMs": 208280,
    "text": "and speaking a common language with the"
  },
  {
    "startMs": 209720,
    "text": "user? And then the second phase to this"
  },
  {
    "startMs": 211760,
    "text": "is also coming up going into the"
  },
  {
    "startMs": 214239,
    "text": "background and doing the heavy lifting"
  },
  {
    "startMs": 215640,
    "text": "for the user, so taking the burden away"
  },
  {
    "startMs": 217440,
    "text": "from the user to describing what they"
  },
  {
    "startMs": 219440,
    "text": "want, and rather going into the"
  },
  {
    "startMs": 220600,
    "text": "background and doing all the"
  },
  {
    "startMs": 222200,
    "text": "comparisons, trade-offs,"
  },
  {
    "startMs": 224280,
    "text": "summarization of all the information"
  },
  {
    "startMs": 226000,
    "text": "that they're looking for, and coming"
  },
  {
    "startMs": 227600,
    "text": "back with the right set of potential"
  },
  {
    "startMs": 229560,
    "text": "options for the user. And then the third"
  },
  {
    "startMs": 231880,
    "text": "phase, once um"
  },
  {
    "startMs": 233920,
    "text": "the user and the agent is ready to go"
  },
  {
    "startMs": 235920,
    "text": "into some kind of a"
  },
  {
    "startMs": 238600,
    "text": "is ready to go into the last phase,"
  },
  {
    "startMs": 239920,
    "text": "which is where um"
  },
  {
    "startMs": 241440,
    "text": "the agent is ready to give out a"
  },
  {
    "startMs": 242720,
    "text": "response. This is where the agent needs"
  },
  {
    "startMs": 245000,
    "text": "to start adapting the response in a way"
  },
  {
    "startMs": 247160,
    "text": "that is most useful for the user. So,"
  },
  {
    "startMs": 249600,
    "text": "typically a lot of systems fail here"
  },
  {
    "startMs": 251240,
    "text": "where they just give out a text-heavy"
  },
  {
    "startMs": 252480,
    "text": "response. What the agent should rather"
  },
  {
    "startMs": 254520,
    "text": "be doing is adapting to the query that"
  },
  {
    "startMs": 256560,
    "text": "the user had in mind. So, should it be"
  },
  {
    "startMs": 258560,
    "text": "using um some kind of a bulleted list?"
  },
  {
    "startMs": 260440,
    "text": "Should it be using comparison tables?"
  },
  {
    "startMs": 262240,
    "text": "Should it be using um visual boards? So,"
  },
  {
    "startMs": 264880,
    "text": "this is where the there's adaptive"
  },
  {
    "startMs": 266880,
    "text": "response happening as well, where the uh"
  },
  {
    "startMs": 268960,
    "text": "agent starts to develop uh more more of"
  },
  {
    "startMs": 271919,
    "text": "a uh smarter uh presentation skills uh"
  },
  {
    "startMs": 275080,
    "text": "for the user to really find the answer"
  },
  {
    "startMs": 276840,
    "text": "that they're looking for. So, we'll be"
  },
  {
    "startMs": 278200,
    "text": "diving into all these topics as I uh"
  },
  {
    "startMs": 280720,
    "text": "kind of work through the presentation."
  },
  {
    "startMs": 283680,
    "text": "So, first and foremost, we're looking at"
  },
  {
    "startMs": 285560,
    "text": "discovery. Like I was mentioning, this"
  },
  {
    "startMs": 287280,
    "text": "is where uh the the the agent needs to"
  },
  {
    "startMs": 290520,
    "text": "remember what exactly matters."
  },
  {
    "startMs": 292840,
    "text": "Um you the agent starts to look at the"
  },
  {
    "startMs": 295120,
    "text": "context across a bunch of different"
  },
  {
    "startMs": 296480,
    "text": "signals. It starts to look at past"
  },
  {
    "startMs": 298000,
    "text": "conversations. It looks at some of the"
  },
  {
    "startMs": 300080,
    "text": "reference images or reference links that"
  },
  {
    "startMs": 301960,
    "text": "the user might have provided. It also"
  },
  {
    "startMs": 303360,
    "text": "starts to look at personal context and"
  },
  {
    "startMs": 305760,
    "text": "starts to build out a working state. So,"
  },
  {
    "startMs": 308000,
    "text": "if you look at the sample code that we"
  },
  {
    "startMs": 309680,
    "text": "have here, like there is there is a goal"
  },
  {
    "startMs": 311480,
    "text": "uh one of the let's let's work with this"
  },
  {
    "startMs": 312840,
    "text": "query where the user is trying to redo"
  },
  {
    "startMs": 314560,
    "text": "their living room with a certain budget"
  },
  {
    "startMs": 316640,
    "text": "in mind. And some of the um some of the"
  },
  {
    "startMs": 319680,
    "text": "things that the agent develops as part"
  },
  {
    "startMs": 321280,
    "text": "of the working state is the session"
  },
  {
    "startMs": 322720,
    "text": "history. It has a user context. It also"
  },
  {
    "startMs": 325480,
    "text": "kind of extracts out the hard constraint"
  },
  {
    "startMs": 327440,
    "text": "that the user might have provided in the"
  },
  {
    "startMs": 329400,
    "text": "query."
  },
  {
    "startMs": 330560,
    "text": "But, things start getting getting"
  },
  {
    "startMs": 331640,
    "text": "interesting when we come to the softer"
  },
  {
    "startMs": 333280,
    "text": "constraints."
  },
  {
    "startMs": 334360,
    "text": "So, this is where the user may not be"
  },
  {
    "startMs": 336240,
    "text": "able to describe what they're looking"
  },
  {
    "startMs": 337520,
    "text": "for and may might have provided like a"
  },
  {
    "startMs": 339440,
    "text": "reference image on, you know, an"
  },
  {
    "startMs": 341480,
    "text": "inspiration that they had in mind or"
  },
  {
    "startMs": 343600,
    "text": "some kind of a layout design that they"
  },
  {
    "startMs": 345040,
    "text": "really liked and is trying to get to the"
  },
  {
    "startMs": 346840,
    "text": "agent in terms of uh this is what speaks"
  },
  {
    "startMs": 349080,
    "text": "more to me. So, this is where the agent"
  },
  {
    "startMs": 350920,
    "text": "starts to be more proactive and pulls"
  },
  {
    "startMs": 352600,
    "text": "out some of the salient signals from the"
  },
  {
    "startMs": 355040,
    "text": "from the reference images and starts to"
  },
  {
    "startMs": 357040,
    "text": "develop a mental model of what the user"
  },
  {
    "startMs": 359120,
    "text": "might really be looking to get at. So,"
  },
  {
    "startMs": 361120,
    "text": "here the user is uh so the agent has"
  },
  {
    "startMs": 363640,
    "text": "specified uh has identified that maybe"
  },
  {
    "startMs": 365480,
    "text": "the style that is working out well for"
  },
  {
    "startMs": 367600,
    "text": "them is of a certain kind. It also uh is"
  },
  {
    "startMs": 370760,
    "text": "also is also starting to work towards a"
  },
  {
    "startMs": 373560,
    "text": "confidence score on like how confident"
  },
  {
    "startMs": 375400,
    "text": "it is in terms of pulling out some of"
  },
  {
    "startMs": 377160,
    "text": "this information present in the"
  },
  {
    "startMs": 379000,
    "text": "multimodal inputs provided. And then the"
  },
  {
    "startMs": 380960,
    "text": "last thing uh that happens as part of"
  },
  {
    "startMs": 383080,
    "text": "developing this working state is also"
  },
  {
    "startMs": 384640,
    "text": "figuring out what are the variables that"
  },
  {
    "startMs": 386880,
    "text": "the agent needs to pull out almost in"
  },
  {
    "startMs": 388800,
    "text": "real time because these variables"
  },
  {
    "startMs": 391240,
    "text": "uh this variables do affect how the"
  },
  {
    "startMs": 393360,
    "text": "results will be displayed back to the"
  },
  {
    "startMs": 394600,
    "text": "user. This could be variables that need"
  },
  {
    "startMs": 396040,
    "text": "to be refreshed in real time like uh"
  },
  {
    "startMs": 398280,
    "text": "inventory because if if what you're"
  },
  {
    "startMs": 400200,
    "text": "providing back to the user is stale"
  },
  {
    "startMs": 402160,
    "text": "information, then it's kind of a moot"
  },
  {
    "startMs": 403919,
    "text": "point. So, these are the uh these are"
  },
  {
    "startMs": 405560,
    "text": "the variables that you want to refresh"
  },
  {
    "startMs": 407360,
    "text": "in real time and is and becomes a part"
  },
  {
    "startMs": 409680,
    "text": "of the agent's working state."
  },
  {
    "startMs": 411600,
    "text": "Um"
  },
  {
    "startMs": 412680,
    "text": "few ways that you can evaluate this"
  },
  {
    "startMs": 414120,
    "text": "state uh the way we have developed our"
  },
  {
    "startMs": 415840,
    "text": "auto raters, we we do make sure that all"
  },
  {
    "startMs": 418480,
    "text": "facts are retained meaning that whatever"
  },
  {
    "startMs": 421120,
    "text": "was mentioned in the part of uh whatever"
  },
  {
    "startMs": 423560,
    "text": "was mentioned in part of the context is"
  },
  {
    "startMs": 426200,
    "text": "properly represented in the working"
  },
  {
    "startMs": 428440,
    "text": "state. We also capture if the confidence"
  },
  {
    "startMs": 431919,
    "text": "collaboration was within a certain error"
  },
  {
    "startMs": 434120,
    "text": "bound because uh the agent needs to be"
  },
  {
    "startMs": 436600,
    "text": "able to confidently pull out these"
  },
  {
    "startMs": 438120,
    "text": "signals from the input. We also focus on"
  },
  {
    "startMs": 440960,
    "text": "getting out um the counterfactual"
  },
  {
    "startMs": 442840,
    "text": "sensitivity uh counterfactual"
  },
  {
    "startMs": 444200,
    "text": "sensitivity. So, we do this by flipping"
  },
  {
    "startMs": 446400,
    "text": "some parts of the queries and making"
  },
  {
    "startMs": 448040,
    "text": "sure that when the query changes, the"
  },
  {
    "startMs": 450560,
    "text": "underlying constraints pulled out by the"
  },
  {
    "startMs": 452240,
    "text": "agent the those also change and the ones"
  },
  {
    "startMs": 454480,
    "text": "that are not relevant stay the same. So,"
  },
  {
    "startMs": 456120,
    "text": "we kind of want to measure the"
  },
  {
    "startMs": 457680,
    "text": "sensitivity both ways."
  },
  {
    "startMs": 460080,
    "text": "And then the second part that happens in"
  },
  {
    "startMs": 461560,
    "text": "the discovery phase as well is once the"
  },
  {
    "startMs": 464240,
    "text": "agent knows what they have what they"
  },
  {
    "startMs": 466280,
    "text": "already know about the user what more do"
  },
  {
    "startMs": 467880,
    "text": "they need to know about the user. So"
  },
  {
    "startMs": 469680,
    "text": "what we call this what we call this is"
  },
  {
    "startMs": 471880,
    "text": "kind of discovering the intent gap."
  },
  {
    "startMs": 475000,
    "text": "There's a lot of times that there are a"
  },
  {
    "startMs": 476880,
    "text": "lot of unknown variables before the"
  },
  {
    "startMs": 478880,
    "text": "agent can provide the best answer and"
  },
  {
    "startMs": 481520,
    "text": "amongst all these unknown variables the"
  },
  {
    "startMs": 483240,
    "text": "agent doesn't need to go and find all"
  },
  {
    "startMs": 484800,
    "text": "the unknown variables up front. So what"
  },
  {
    "startMs": 487200,
    "text": "I mean by that is in this unknown in in"
  },
  {
    "startMs": 489240,
    "text": "our working example some of the unknown"
  },
  {
    "startMs": 490920,
    "text": "variables that the agent said they need"
  },
  {
    "startMs": 492800,
    "text": "to find out more about is maybe knowing"
  },
  {
    "startMs": 495440,
    "text": "what the room width would be for the"
  },
  {
    "startMs": 496880,
    "text": "user or even kind of working on"
  },
  {
    "startMs": 499480,
    "text": "improving the confidence for the style"
  },
  {
    "startMs": 501200,
    "text": "before they can recommend back the"
  },
  {
    "startMs": 502240,
    "text": "results. Now"
  },
  {
    "startMs": 504520,
    "text": "once these unknown variables are figured"
  },
  {
    "startMs": 506400,
    "text": "out the agent also needs to work on a"
  },
  {
    "startMs": 508480,
    "text": "collaborative strategy compare all the"
  },
  {
    "startMs": 510120,
    "text": "different moves possible and then figure"
  },
  {
    "startMs": 512200,
    "text": "out what is that one unknown variable"
  },
  {
    "startMs": 514080,
    "text": "that it should prioritize such that it"
  },
  {
    "startMs": 515680,
    "text": "has the maximal information gain at that"
  },
  {
    "startMs": 517320,
    "text": "point. So in this case"
  },
  {
    "startMs": 519919,
    "text": "I mean one could argue that maybe"
  },
  {
    "startMs": 521280,
    "text": "finding out the room width is the best"
  },
  {
    "startMs": 523159,
    "text": "next move for the agent because if the"
  },
  {
    "startMs": 525839,
    "text": "products that the the agent is"
  },
  {
    "startMs": 527560,
    "text": "recommending doesn't fit into the room"
  },
  {
    "startMs": 529240,
    "text": "then again it's a moot point and that is"
  },
  {
    "startMs": 531320,
    "text": "one variable that is going to"
  },
  {
    "startMs": 532320,
    "text": "meaningfully change the"
  },
  {
    "startMs": 534320,
    "text": "the direction of the conversation and"
  },
  {
    "startMs": 536360,
    "text": "that's what the agent works on in this"
  },
  {
    "startMs": 537640,
    "text": "step kind of figuring out what's the"
  },
  {
    "startMs": 538960,
    "text": "best next thing to ask to the user and"
  },
  {
    "startMs": 541560,
    "text": "what's why is that the best next thing"
  },
  {
    "startMs": 543920,
    "text": "to ask as well. Once it has okay so I'll"
  },
  {
    "startMs": 546360,
    "text": "go ahead into the auto data section on"
  },
  {
    "startMs": 548040,
    "text": "like why how do we evaluate this"
  },
  {
    "startMs": 550560,
    "text": "collaborative strategy we first focus on"
  },
  {
    "startMs": 553120,
    "text": "making sure that the agent is able to"
  },
  {
    "startMs": 555560,
    "text": "identify all the different blockers that"
  },
  {
    "startMs": 557520,
    "text": "are needed to be answered before the"
  },
  {
    "startMs": 559720,
    "text": "agent can come back with meaningful"
  },
  {
    "startMs": 561839,
    "text": "responses. We also work towards making"
  },
  {
    "startMs": 564520,
    "text": "sure that the agent is um"
  },
  {
    "startMs": 566680,
    "text": "optimal in trying to get some of these"
  },
  {
    "startMs": 568600,
    "text": "responses so we also don't want to have"
  },
  {
    "startMs": 570440,
    "text": "the agent constantly going into the loop"
  },
  {
    "startMs": 571880,
    "text": "and continually continuously asking"
  },
  {
    "startMs": 573520,
    "text": "these questions so over asking is"
  },
  {
    "startMs": 575360,
    "text": "definitely something we flag. We also"
  },
  {
    "startMs": 577520,
    "text": "work on question utility. So, again, how"
  },
  {
    "startMs": 581040,
    "text": "optimally is the question being asked?"
  },
  {
    "startMs": 582920,
    "text": "Is the question indeed useful to get the"
  },
  {
    "startMs": 585440,
    "text": "right or elicit the right preference"
  },
  {
    "startMs": 587280,
    "text": "from the user? And many more."
  },
  {
    "startMs": 591160,
    "text": "Okay."
  },
  {
    "startMs": 592280,
    "text": "Moving on. The The second part that I"
  },
  {
    "startMs": 594360,
    "text": "was mentioning is the multimodal"
  },
  {
    "startMs": 596680,
    "text": "elicitation. This is where the research"
  },
  {
    "startMs": 599400,
    "text": "phase happens."
  },
  {
    "startMs": 601040,
    "text": "So, play along, but like let's say the"
  },
  {
    "startMs": 603640,
    "text": "the agent has asked about the room"
  },
  {
    "startMs": 605680,
    "text": "width, the user has provided the"
  },
  {
    "startMs": 607920,
    "text": "response, and then the agent goes to the"
  },
  {
    "startMs": 610200,
    "text": "next step, which is where the agent is"
  },
  {
    "startMs": 612000,
    "text": "asking about the next constraint that it"
  },
  {
    "startMs": 614320,
    "text": "needs to know about is what was the"
  },
  {
    "startMs": 615760,
    "text": "style preference of the user."
  },
  {
    "startMs": 617760,
    "text": "Now, the first first thing that the"
  },
  {
    "startMs": 620200,
    "text": "agent needs to do is kind of form this"
  },
  {
    "startMs": 622040,
    "text": "temporary bridge between the constraint"
  },
  {
    "startMs": 624440,
    "text": "itself and how that maps back to the"
  },
  {
    "startMs": 627760,
    "text": "the product catalog and the ontology in"
  },
  {
    "startMs": 629720,
    "text": "your knowledge knowledge database. And"
  },
  {
    "startMs": 631200,
    "text": "this is going to be important because"
  },
  {
    "startMs": 632440,
    "text": "when you start retrieving these"
  },
  {
    "startMs": 633480,
    "text": "products, you want to have a way to map"
  },
  {
    "startMs": 635600,
    "text": "these constraints back to your knowledge"
  },
  {
    "startMs": 637800,
    "text": "base. So, this almost happens in real"
  },
  {
    "startMs": 639720,
    "text": "time where we map the known constraint"
  },
  {
    "startMs": 643040,
    "text": "or the constraint that the agent is"
  },
  {
    "startMs": 645280,
    "text": "exploring back to the knowledge"
  },
  {
    "startMs": 647320,
    "text": "database. We also work towards um"
  },
  {
    "startMs": 650960,
    "text": "figuring out We also work on like the"
  },
  {
    "startMs": 653000,
    "text": "agent knowing what is the best way to"
  },
  {
    "startMs": 655600,
    "text": "get the response for this constraint as"
  },
  {
    "startMs": 657640,
    "text": "well. So, in this case, the agent has"
  },
  {
    "startMs": 660120,
    "text": "decided that maybe since this is this"
  },
  {
    "startMs": 662560,
    "text": "kind of a subjective constraint, the"
  },
  {
    "startMs": 664680,
    "text": "textual the actual elicitation is not"
  },
  {
    "startMs": 666320,
    "text": "the best way to do this. So, one of the"
  },
  {
    "startMs": 668400,
    "text": "ways that the agent thinks this could"
  },
  {
    "startMs": 670280,
    "text": "this could be elicited back from the"
  },
  {
    "startMs": 672880,
    "text": "user is using some kind of a visual"
  },
  {
    "startMs": 675280,
    "text": "preference"
  },
  {
    "startMs": 676520,
    "text": "board."
  },
  {
    "startMs": 677560,
    "text": "So, the agent then goes back to"
  },
  {
    "startMs": 679839,
    "text": "determining what is the best form of"
  },
  {
    "startMs": 681640,
    "text": "options to show to the user. This could"
  },
  {
    "startMs": 683760,
    "text": "be a combination of figuring out from"
  },
  {
    "startMs": 686400,
    "text": "the existing constraints, the past"
  },
  {
    "startMs": 688200,
    "text": "conversation, and then the temporary"
  },
  {
    "startMs": 690000,
    "text": "mapping that you've created from the"
  },
  {
    "startMs": 691560,
    "text": "constraints back to your product"
  },
  {
    "startMs": 692920,
    "text": "ontology. So, in this case, the the"
  },
  {
    "startMs": 695680,
    "text": "agent thinks that maybe coming up with a"
  },
  {
    "startMs": 697280,
    "text": "few styles that are most similar to what"
  },
  {
    "startMs": 699880,
    "text": "the user had provided as reference image"
  },
  {
    "startMs": 701800,
    "text": "could be a good way to start thinking or"
  },
  {
    "startMs": 703480,
    "text": "guiding the user towards a common"
  },
  {
    "startMs": 704920,
    "text": "language on what could be uh something"
  },
  {
    "startMs": 707600,
    "text": "that the user is interested in."
  },
  {
    "startMs": 709720,
    "text": "And then the the then the agent also"
  },
  {
    "startMs": 713280,
    "text": "goes into kind of um observing the space"
  },
  {
    "startMs": 717000,
    "text": "of what kind of reactions the user is"
  },
  {
    "startMs": 719000,
    "text": "giving. So, it could start looking at"
  },
  {
    "startMs": 720920,
    "text": "these micro signals of if there was a"
  },
  {
    "startMs": 722400,
    "text": "hover or a click in a certain direction"
  },
  {
    "startMs": 724160,
    "text": "and starts updating its its confidence"
  },
  {
    "startMs": 726160,
    "text": "model on what kind of signals uh what"
  },
  {
    "startMs": 728920,
    "text": "kind of signals can be used to im-"
  },
  {
    "startMs": 730560,
    "text": "improve the confidence in like what"
  },
  {
    "startMs": 732160,
    "text": "could be the style preference for the"
  },
  {
    "startMs": 733720,
    "text": "user."
  },
  {
    "startMs": 734920,
    "text": "Um some of the auto raters we use here,"
  },
  {
    "startMs": 737720,
    "text": "we do look at how efficient the how"
  },
  {
    "startMs": 740720,
    "text": "efficient the agent is in discovering"
  },
  {
    "startMs": 742320,
    "text": "hidden preferences. So, typically, we"
  },
  {
    "startMs": 743840,
    "text": "would use a user simulator, feed it with"
  },
  {
    "startMs": 746240,
    "text": "some constraints, and then we'll see how"
  },
  {
    "startMs": 748480,
    "text": "how efficient the agent was in kind of"
  },
  {
    "startMs": 750400,
    "text": "eliciting some of these constraints. We"
  },
  {
    "startMs": 752400,
    "text": "also focus on turn efficiency. So,"
  },
  {
    "startMs": 755040,
    "text": "how efficient was the user in how many"
  },
  {
    "startMs": 757440,
    "text": "turns did it take for the user to be"
  },
  {
    "startMs": 758960,
    "text": "able to to be able to uncover all these"
  },
  {
    "startMs": 760920,
    "text": "hidden preferences. Ideally, we don't"
  },
  {
    "startMs": 762680,
    "text": "want the user to go into this loop and"
  },
  {
    "startMs": 764040,
    "text": "keep asking the same questions again and"
  },
  {
    "startMs": 765680,
    "text": "again and again. Or also, we don't want"
  },
  {
    "startMs": 767000,
    "text": "to go into the loop of asking some"
  },
  {
    "startMs": 769600,
    "text": "um some questions which may not give you"
  },
  {
    "startMs": 771120,
    "text": "the best uh which may not give you the"
  },
  {
    "startMs": 773080,
    "text": "best information required to proceed the"
  },
  {
    "startMs": 775480,
    "text": "conversation. And then we also look at"
  },
  {
    "startMs": 777480,
    "text": "forma- format selection accuracy. So, we"
  },
  {
    "startMs": 779240,
    "text": "typically also look for if the agent is"
  },
  {
    "startMs": 781880,
    "text": "asking the right right question in the"
  },
  {
    "startMs": 783680,
    "text": "right format. So, for example, if the if"
  },
  {
    "startMs": 786440,
    "text": "the question was something that was"
  },
  {
    "startMs": 787760,
    "text": "easily statable, the right format could"
  },
  {
    "startMs": 789520,
    "text": "be a textual elicitation. But if if this"
  },
  {
    "startMs": 792040,
    "text": "was more of a"
  },
  {
    "startMs": 793560,
    "text": "fuzzy question where the user is clearly"
  },
  {
    "startMs": 795400,
    "text": "having an articulation gap and is not"
  },
  {
    "startMs": 796920,
    "text": "able to describe the preference, maybe"
  },
  {
    "startMs": 799280,
    "text": "the best way to do this is speak a"
  },
  {
    "startMs": 800560,
    "text": "common language and come up with some"
  },
  {
    "startMs": 802560,
    "text": "visual anchor points for the user to say"
  },
  {
    "startMs": 805760,
    "text": "what speaks more to them."
  },
  {
    "startMs": 809480,
    "text": "Um"
  },
  {
    "startMs": 811000,
    "text": "Yeah, so then the last step in this"
  },
  {
    "startMs": 812480,
    "text": "process, once the So, to recap like"
  },
  {
    "startMs": 815160,
    "text": "basically the Now the agent knows"
  },
  {
    "startMs": 816680,
    "text": "exactly what they what the user is"
  },
  {
    "startMs": 818880,
    "text": "looking for. The agent has identified a"
  },
  {
    "startMs": 820880,
    "text": "collaboration strategy, has figured out"
  },
  {
    "startMs": 822440,
    "text": "all the preferences for the user, and"
  },
  {
    "startMs": 824200,
    "text": "knows what is going to be most optimal"
  },
  {
    "startMs": 826280,
    "text": "in terms of um the different priorities"
  },
  {
    "startMs": 828280,
    "text": "for the user. The last step in this"
  },
  {
    "startMs": 830280,
    "text": "process is to also use the model"
  },
  {
    "startMs": 832240,
    "text": "intelligence to figure out what is the"
  },
  {
    "startMs": 833640,
    "text": "best way to provide this response back"
  },
  {
    "startMs": 835360,
    "text": "to the user. So, in this working"
  },
  {
    "startMs": 837960,
    "text": "example, we found out that the agent"
  },
  {
    "startMs": 840760,
    "text": "knows what the style preference is, has"
  },
  {
    "startMs": 842320,
    "text": "figured out that the user is looking to"
  },
  {
    "startMs": 844960,
    "text": "buy products under a certain budget, and"
  },
  {
    "startMs": 847280,
    "text": "knows what are the different dimensions"
  },
  {
    "startMs": 848760,
    "text": "along which it needs to find uh these"
  },
  {
    "startMs": 850600,
    "text": "products for because through the"
  },
  {
    "startMs": 851760,
    "text": "conversation they figured out some of"
  },
  {
    "startMs": 853280,
    "text": "the different um"
  },
  {
    "startMs": 855680,
    "text": "different metadata information that is"
  },
  {
    "startMs": 856920,
    "text": "going to be relevant to surface when"
  },
  {
    "startMs": 858320,
    "text": "they're giving out this product"
  },
  {
    "startMs": 859400,
    "text": "information back to the user."
  },
  {
    "startMs": 861200,
    "text": "So, the one one important step that"
  },
  {
    "startMs": 863520,
    "text": "happens at this this stage is figuring"
  },
  {
    "startMs": 865360,
    "text": "out what's the best way to surface back"
  },
  {
    "startMs": 867440,
    "text": "this response. So, for example, like if"
  },
  {
    "startMs": 869320,
    "text": "the user was looking for a particular"
  },
  {
    "startMs": 872000,
    "text": "policy or review information about a"
  },
  {
    "startMs": 874200,
    "text": "particular product, maybe the best way"
  },
  {
    "startMs": 875760,
    "text": "to do this is to go give out a summary"
  },
  {
    "startMs": 878320,
    "text": "or a bulleted list. But if the if the"
  },
  {
    "startMs": 881000,
    "text": "user was looking more towards comparing"
  },
  {
    "startMs": 882800,
    "text": "two different products, maybe the best"
  },
  {
    "startMs": 884320,
    "text": "response is to give out a trade-off"
  },
  {
    "startMs": 886280,
    "text": "table or a comparison table across the"
  },
  {
    "startMs": 887680,
    "text": "different axis that the user cares"
  },
  {
    "startMs": 889360,
    "text": "about."
  },
  {
    "startMs": 890600,
    "text": "Um and in case that in our example where"
  },
  {
    "startMs": 892839,
    "text": "the user was looking more towards kind"
  },
  {
    "startMs": 894520,
    "text": "of style inspiration or ideas on how"
  },
  {
    "startMs": 896520,
    "text": "they can kind of redo their room, then"
  },
  {
    "startMs": 898640,
    "text": "maybe the best way is to give out some"
  },
  {
    "startMs": 900280,
    "text": "visual references and product"
  },
  {
    "startMs": 901960,
    "text": "inspiration photos on like how What are"
  },
  {
    "startMs": 903920,
    "text": "the different options and possibilities"
  },
  {
    "startMs": 905800,
    "text": "uh for the user to uncover. So,"
  },
  {
    "startMs": 908520,
    "text": "some other ways that we focus on"
  },
  {
    "startMs": 910760,
    "text": "evaluating uh this stage is focusing on"
  },
  {
    "startMs": 913440,
    "text": "the format accuracy. So, we want to make"
  },
  {
    "startMs": 915040,
    "text": "sure that the response format is kind of"
  },
  {
    "startMs": 918080,
    "text": "optimal for the user query so that they"
  },
  {
    "startMs": 920000,
    "text": "they find out exactly what they're"
  },
  {
    "startMs": 921240,
    "text": "looking for."
  },
  {
    "startMs": 922400,
    "text": "Uh ideally the information that they're"
  },
  {
    "startMs": 923720,
    "text": "looking for should not be buried in the"
  },
  {
    "startMs": 925000,
    "text": "response, but should be easy for the"
  },
  {
    "startMs": 926640,
    "text": "user to spot so they can come into the"
  },
  {
    "startMs": 928240,
    "text": "next stage in the intent journey, which"
  },
  {
    "startMs": 930360,
    "text": "is to basically buy the product. We also"
  },
  {
    "startMs": 933520,
    "text": "look at data fidelity, which is to make"
  },
  {
    "startMs": 935640,
    "text": "sure that the model is not hallucinating"
  },
  {
    "startMs": 937360,
    "text": "and it's really capturing the"
  },
  {
    "startMs": 938960,
    "text": "information in the correct format in the"
  },
  {
    "startMs": 941160,
    "text": "you know, the the information is just"
  },
  {
    "startMs": 942440,
    "text": "accurate and is captured across"
  },
  {
    "startMs": 944920,
    "text": "the response. We also look at user"
  },
  {
    "startMs": 947320,
    "text": "actionability. So, this is ensuring that"
  },
  {
    "startMs": 950000,
    "text": "the response format is such that the"
  },
  {
    "startMs": 951840,
    "text": "user is very confident and and commits"
  },
  {
    "startMs": 954080,
    "text": "to the next action, which is like I"
  },
  {
    "startMs": 956160,
    "text": "said, the action to purchase the"
  },
  {
    "startMs": 957600,
    "text": "product."
  },
  {
    "startMs": 959480,
    "text": "So,"
  },
  {
    "startMs": 961120,
    "text": "just to recap so far, what we have is um"
  },
  {
    "startMs": 964440,
    "text": "you want to design the product. Um"
  },
  {
    "startMs": 967200,
    "text": "you want to you want to design the"
  },
  {
    "startMs": 968200,
    "text": "product such that you're prepared to"
  },
  {
    "startMs": 969400,
    "text": "accept vibes, like I say. So, that is to"
  },
  {
    "startMs": 972160,
    "text": "say that users will come with fuzzy"
  },
  {
    "startMs": 974080,
    "text": "intent. Users will not have a"
  },
  {
    "startMs": 975800,
    "text": "well-defined goal. So, you want to make"
  },
  {
    "startMs": 977240,
    "text": "sure that your system is able to work"
  },
  {
    "startMs": 978840,
    "text": "through queries that are not clean."
  },
  {
    "startMs": 981520,
    "text": "Um the second takeaway is you want to"
  },
  {
    "startMs": 983400,
    "text": "focus on showing and asking rather than"
  },
  {
    "startMs": 985720,
    "text": "always asking with textual textual with"
  },
  {
    "startMs": 988040,
    "text": "textual elicitations. Again, like"
  },
  {
    "startMs": 989920,
    "text": "visuals and comparisons do reveal"
  },
  {
    "startMs": 991640,
    "text": "preferences much much faster. It allows"
  },
  {
    "startMs": 993680,
    "text": "you the agent and the user to speak a"
  },
  {
    "startMs": 995720,
    "text": "common language."
  },
  {
    "startMs": 997520,
    "text": "Third one I would say is shape the"
  },
  {
    "startMs": 999320,
    "text": "answer. So, do focus on making sure that"
  },
  {
    "startMs": 1002200,
    "text": "the presentation format is ideal for the"
  },
  {
    "startMs": 1004960,
    "text": "user being able to find the right"
  },
  {
    "startMs": 1006200,
    "text": "information."
  },
  {
    "startMs": 1007520,
    "text": "Um the way you have the model response"
  },
  {
    "startMs": 1009680,
    "text": "structure is also very much part of the"
  },
  {
    "startMs": 1011920,
    "text": "intelligence. And then the last one is"
  },
  {
    "startMs": 1014400,
    "text": "make sure that you grade the loop. You"
  },
  {
    "startMs": 1016560,
    "text": "have the right auto rater set up on"
  },
  {
    "startMs": 1018680,
    "text": "every step of the process. Um and"
  },
  {
    "startMs": 1021040,
    "text": "honestly, developing these auto raters"
  },
  {
    "startMs": 1022960,
    "text": "is a"
  },
  {
    "startMs": 1024160,
    "text": "is is almost like an evolving system. It"
  },
  {
    "startMs": 1026000,
    "text": "starts very simple, but as and when you"
  },
  {
    "startMs": 1027640,
    "text": "start the the system starts evolving,"
  },
  {
    "startMs": 1029400,
    "text": "you want the auto raters to kind of"
  },
  {
    "startMs": 1031199,
    "text": "gradually grow with your system and"
  },
  {
    "startMs": 1032720,
    "text": "start"
  },
  {
    "startMs": 1033959,
    "text": "um"
  },
  {
    "startMs": 1034640,
    "text": "yeah, it which just gradually grow with"
  },
  {
    "startMs": 1036079,
    "text": "your system."
  },
  {
    "startMs": 1037319,
    "text": "So, that's all I had. Um"
  },
  {
    "startMs": 1039600,
    "text": "I can take a few questions, but"
  },
  {
    "startMs": 1041839,
    "text": "hopefully the learnings we shared are"
  },
  {
    "startMs": 1043800,
    "text": "useful for whatever you folks are"
  },
  {
    "startMs": 1045280,
    "text": "building."
  },
  {
    "startMs": 1046839,
    "text": "Yeah."
  },
  {
    "startMs": 1054520,
    "text": "Yeah."
  },
  {
    "startMs": 1090800,
    "text": "Yeah."
  },
  {
    "startMs": 1092080,
    "text": "Yeah, great question. So, the question"
  },
  {
    "startMs": 1093280,
    "text": "is about what should the how should the"
  },
  {
    "startMs": 1095000,
    "text": "ontology be structured on the merchant"
  },
  {
    "startMs": 1096560,
    "text": "side so that it's fair both for the"
  },
  {
    "startMs": 1098400,
    "text": "agent and the merchants. So, yes, we do"
  },
  {
    "startMs": 1101480,
    "text": "take a lot of advantage on the domain"
  },
  {
    "startMs": 1103720,
    "text": "expertise of the merchant on as to what"
  },
  {
    "startMs": 1105560,
    "text": "they're trying to sell and like we do"
  },
  {
    "startMs": 1107400,
    "text": "work towards creating So, you remember"
  },
  {
    "startMs": 1109000,
    "text": "how I was mentioning about the bridge"
  },
  {
    "startMs": 1110320,
    "text": "between the constraints that the user"
  },
  {
    "startMs": 1112720,
    "text": "might have specified and then what the"
  },
  {
    "startMs": 1114520,
    "text": "agent"
  },
  {
    "startMs": 1115520,
    "text": "kind of understands. That is where we do"
  },
  {
    "startMs": 1117560,
    "text": "expect a lot of intelligence to flow"
  },
  {
    "startMs": 1119120,
    "text": "from the merchant side where the"
  },
  {
    "startMs": 1120560,
    "text": "ontology on how that constraint could"
  },
  {
    "startMs": 1122560,
    "text": "map to the different metadata that the"
  },
  {
    "startMs": 1124080,
    "text": "agent has. Sorry, the that the merchant"
  },
  {
    "startMs": 1125840,
    "text": "has maps in. So, yes, we do partner a"
  },
  {
    "startMs": 1127960,
    "text": "lot and then there's also like the UCP"
  },
  {
    "startMs": 1129880,
    "text": "stuff that we launched recently which"
  },
  {
    "startMs": 1131600,
    "text": "allows all the merchants to kind of"
  },
  {
    "startMs": 1133320,
    "text": "start speaking the common language with"
  },
  {
    "startMs": 1134760,
    "text": "the agent as well."
  },
  {
    "startMs": 1160560,
    "text": ">> Yeah, so ideally, I mean, honestly right"
  },
  {
    "startMs": 1162480,
    "text": "now we focus on making sure all of this"
  },
  {
    "startMs": 1164480,
    "text": "flows back to the agent and the agent"
  },
  {
    "startMs": 1166080,
    "text": "makes the decisions because you want to"
  },
  {
    "startMs": 1167440,
    "text": "build like a horizontal common layer"
  },
  {
    "startMs": 1168840,
    "text": "across all the different merchants. So,"
  },
  {
    "startMs": 1171320,
    "text": "and also like it should be a seamless"
  },
  {
    "startMs": 1172920,
    "text": "experience for the for the user who's"
  },
  {
    "startMs": 1175560,
    "text": "interacting with our apps. So, right now"
  },
  {
    "startMs": 1177840,
    "text": "the response format is very much part of"
  },
  {
    "startMs": 1180520,
    "text": "the agent's intelligence. It is not"
  },
  {
    "startMs": 1182120,
    "text": "something that the merchant gets to"
  },
  {
    "startMs": 1184320,
    "text": "decide."
  },
  {
    "startMs": 1185720,
    "text": "Yeah."
  },
  {
    "startMs": 1186840,
    "text": "Yes."
  },
  {
    "startMs": 1187560,
    "text": ">> I'm just curious your opinion what"
  },
  {
    "startMs": 1189400,
    "text": "happens when your user is not a user"
  },
  {
    "startMs": 1191000,
    "text": "anymore, it's an agent."
  },
  {
    "startMs": 1193160,
    "text": ">> That's a great question."
  },
  {
    "startMs": 1194920,
    "text": "We're in the early stages of building"
  },
  {
    "startMs": 1195960,
    "text": "this out, but I think I mean there could"
  },
  {
    "startMs": 1197680,
    "text": "be a case where um"
  },
  {
    "startMs": 1201040,
    "text": "Yeah."
  },
  {
    "startMs": 1202120,
    "text": "Yeah, yeah, exactly. So, yeah, I would I"
  },
  {
    "startMs": 1203400,
    "text": "would expect like an MCP to be the"
  },
  {
    "startMs": 1205040,
    "text": "interface between the two for sure."
  },
  {
    "startMs": 1207320,
    "text": "Honestly, we haven't gotten to a point"
  },
  {
    "startMs": 1209200,
    "text": "where we have agents interacting with"
  },
  {
    "startMs": 1211000,
    "text": "our agents just yet. Also, like what"
  },
  {
    "startMs": 1212840,
    "text": "we've realized at least from our user"
  },
  {
    "startMs": 1214520,
    "text": "studies is"
  },
  {
    "startMs": 1215880,
    "text": "users really like to be more involved in"
  },
  {
    "startMs": 1217920,
    "text": "the process of choosing or even"
  },
  {
    "startMs": 1219560,
    "text": "exploring the different possibilities."
  },
  {
    "startMs": 1221080,
    "text": "So, during the upper funnel journeys"
  },
  {
    "startMs": 1222680,
    "text": "where users is looking more towards"
  },
  {
    "startMs": 1224400,
    "text": "discovery, inspiration, that is where"
  },
  {
    "startMs": 1226480,
    "text": "they would rather be interacting with"
  },
  {
    "startMs": 1228000,
    "text": "the system than with their agent. I"
  },
  {
    "startMs": 1229600,
    "text": "think where the agent typically comes in"
  },
  {
    "startMs": 1231280,
    "text": "or even where"
  },
  {
    "startMs": 1232840,
    "text": "what we've heard is like the was the"
  },
  {
    "startMs": 1234200,
    "text": "lower end of the journey where they're"
  },
  {
    "startMs": 1235680,
    "text": "just looking to compare or negotiate or"
  },
  {
    "startMs": 1237640,
    "text": "compare prices across different"
  },
  {
    "startMs": 1238800,
    "text": "merchants, but very much up there in the"
  },
  {
    "startMs": 1240240,
    "text": "funnel, it's the users who kind of"
  },
  {
    "startMs": 1242160,
    "text": "interact more with our systems. So,"
  },
  {
    "startMs": 1243560,
    "text": "yeah."
  },
  {
    "startMs": 1246080,
    "text": "Yeah, I think"
  },
  {
    "startMs": 1247360,
    "text": "yeah, I can take questions outside, but"
  },
  {
    "startMs": 1248840,
    "text": "thank you folks for coming."
  },
  {
    "startMs": 1265474,
    "text": ">> [music]"
  }
]