[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12880,
    "text": "Hi everyone. How's it going? Hey"
  },
  {
    "startMs": 15440,
    "text": "Patricia, how are you?"
  },
  {
    "startMs": 17440,
    "text": ">> Uh so last presentation of the day, so"
  },
  {
    "startMs": 20320,
    "text": "let's make it count. Um,"
  },
  {
    "startMs": 23359,
    "text": "all right. Let's, uh, let me start with"
  },
  {
    "startMs": 25760,
    "text": "a little bit of background on myself."
  },
  {
    "startMs": 28560,
    "text": "And, um, my background, I'm a voice"
  },
  {
    "startMs": 33120,
    "text": "subject matter expert. I've been working"
  },
  {
    "startMs": 35200,
    "text": "in voice AI for a long time across"
  },
  {
    "startMs": 37680,
    "text": "different surfaces, devices, and um,"
  },
  {
    "startMs": 42719,
    "text": "both at Alexa, at at Roku, at my own"
  },
  {
    "startMs": 46239,
    "text": "startups, you know, in the app store."
  },
  {
    "startMs": 49120,
    "text": "And my perspective is a little different"
  },
  {
    "startMs": 52079,
    "text": "from a lot of other voice AI"
  },
  {
    "startMs": 55440,
    "text": "practitioners. I think it's a"
  },
  {
    "startMs": 58399,
    "text": "combination of um a deep um voice user"
  },
  {
    "startMs": 62559,
    "text": "interface expertise and intu intuition"
  },
  {
    "startMs": 65840,
    "text": "mixed in with new technical approaches"
  },
  {
    "startMs": 69040,
    "text": "uh that I think can produce really"
  },
  {
    "startMs": 71119,
    "text": "magical experiences. So I think it's"
  },
  {
    "startMs": 72960,
    "text": "both sides and I think that's especially"
  },
  {
    "startMs": 75119,
    "text": "true in this new area that we're in with"
  },
  {
    "startMs": 77759,
    "text": "frontier tech where the human interface"
  },
  {
    "startMs": 80720,
    "text": "is basically being redefined. So let me"
  },
  {
    "startMs": 84479,
    "text": "start with uh I'll just blast through"
  },
  {
    "startMs": 86720,
    "text": "the first couple of slides then get to"
  },
  {
    "startMs": 88159,
    "text": "the premise. I think everybody knows"
  },
  {
    "startMs": 90080,
    "text": "that voice has incredible potential."
  },
  {
    "startMs": 92960,
    "text": "There's the power of voice I think"
  },
  {
    "startMs": 95119,
    "text": "across everywhere. It's the most natural"
  },
  {
    "startMs": 97360,
    "text": "interface. Humans love talking. And uh"
  },
  {
    "startMs": 100799,
    "text": "the problem is the other half is the"
  },
  {
    "startMs": 103040,
    "text": "pain of voice. So it's the power and the"
  },
  {
    "startMs": 104960,
    "text": "pain. Voice is errorprone. And I think"
  },
  {
    "startMs": 109200,
    "text": "those errors are going to continue for a"
  },
  {
    "startMs": 112000,
    "text": "while. And I think the cost or"
  },
  {
    "startMs": 114560,
    "text": "consequence of those errors is going to"
  },
  {
    "startMs": 116720,
    "text": "grow, especially as we go fromational"
  },
  {
    "startMs": 119920,
    "text": "AI bots to embodied AI where rather than"
  },
  {
    "startMs": 123680,
    "text": "just giving answers that might be"
  },
  {
    "startMs": 125680,
    "text": "erroneous,"
  },
  {
    "startMs": 127200,
    "text": "we're going to have AI systems take"
  },
  {
    "startMs": 129840,
    "text": "physical actions or digital actions"
  },
  {
    "startMs": 132800,
    "text": "where, you know, if the robot throws"
  },
  {
    "startMs": 135440,
    "text": "your watch out with the trash, it's a"
  },
  {
    "startMs": 137680,
    "text": "lot worse than playing the wrong song."
  },
  {
    "startMs": 139760,
    "text": "So I do think that a new approach is"
  },
  {
    "startMs": 143599,
    "text": "definitely needed and here's the TLDDR"
  },
  {
    "startMs": 146879,
    "text": "of the premise we're going to walk"
  },
  {
    "startMs": 148000,
    "text": "through today. Um there are two ways to"
  },
  {
    "startMs": 151360,
    "text": "improve customer or user satisfaction of"
  },
  {
    "startMs": 155120,
    "text": "a voice AI assistant and that is by"
  },
  {
    "startMs": 158000,
    "text": "increasing accuracy which people know"
  },
  {
    "startMs": 159920,
    "text": "about I mean technically accuracy and"
  },
  {
    "startMs": 163680,
    "text": "the other is a different knob that we"
  },
  {
    "startMs": 166319,
    "text": "have that we are not using adequately"
  },
  {
    "startMs": 168640,
    "text": "and I'll call that a system decision"
  },
  {
    "startMs": 171599,
    "text": "which we will define which is orthogonal"
  },
  {
    "startMs": 174319,
    "text": "which is different from accuracy and I"
  },
  {
    "startMs": 177519,
    "text": "believe This approach which I have used"
  },
  {
    "startMs": 181760,
    "text": "in several different environments and"
  },
  {
    "startMs": 184239,
    "text": "seen some success I think is a promising"
  },
  {
    "startMs": 187200,
    "text": "area that we should consider developing."
  },
  {
    "startMs": 190159,
    "text": "Um let me walk through this with a"
  },
  {
    "startMs": 193280,
    "text": "simple smart speaker example and we'll"
  },
  {
    "startMs": 196640,
    "text": "go step by step with this approach but"
  },
  {
    "startMs": 199360,
    "text": "it is a scalable approach that I think"
  },
  {
    "startMs": 202400,
    "text": "uh can apply across different surfaces"
  },
  {
    "startMs": 204480,
    "text": "and devices. So let's get started. So"
  },
  {
    "startMs": 207680,
    "text": "suppose we all you know are making a"
  },
  {
    "startMs": 210400,
    "text": "smart speaker coincidentally called uh"
  },
  {
    "startMs": 213760,
    "text": "Alexa and Alexa is very simple. It just"
  },
  {
    "startMs": 217920,
    "text": "allows you to you know ask for music and"
  },
  {
    "startMs": 220159,
    "text": "it'll play a song and of course it will"
  },
  {
    "startMs": 223280,
    "text": "play either the song you wanted or a"
  },
  {
    "startMs": 225599,
    "text": "different song. So it'll be right or"
  },
  {
    "startMs": 227360,
    "text": "it'll be wrong. This isn't that"
  },
  {
    "startMs": 229599,
    "text": "different from what you've seen out"
  },
  {
    "startMs": 231440,
    "text": "there. Um now let's to first talk about"
  },
  {
    "startMs": 234799,
    "text": "accuracy. Accuracy. Let's say we define"
  },
  {
    "startMs": 237760,
    "text": "it as we you know take a thousand spoken"
  },
  {
    "startMs": 239840,
    "text": "requests. We observe the input and the"
  },
  {
    "startMs": 242480,
    "text": "output. We label it and we look at this."
  },
  {
    "startMs": 245439,
    "text": "This is the map of a thousand points and"
  },
  {
    "startMs": 248000,
    "text": "79% of the time 790 dots here were"
  },
  {
    "startMs": 252640,
    "text": "actually the correct song. This is let's"
  },
  {
    "startMs": 254400,
    "text": "say human annotated 20% 21% wrong song."
  },
  {
    "startMs": 259359,
    "text": "So that's the accuracy. Now, like I"
  },
  {
    "startMs": 262160,
    "text": "said, knob one is to spend a lot of time"
  },
  {
    "startMs": 265520,
    "text": "working on improving the accuracy, you"
  },
  {
    "startMs": 268400,
    "text": "know, um, percentage point by percentage"
  },
  {
    "startMs": 270960,
    "text": "point at any layer in the stack."
  },
  {
    "startMs": 273040,
    "text": "There's, if it's a cascaded system, you"
  },
  {
    "startMs": 275360,
    "text": "know, there's a perhaps a wakeword layer"
  },
  {
    "startMs": 278080,
    "text": "and a speech ASR layer and a NLU layer"
  },
  {
    "startMs": 281040,
    "text": "which might have intent classification,"
  },
  {
    "startMs": 284160,
    "text": "entity extraction, a lot of different"
  },
  {
    "startMs": 285840,
    "text": "layers, VAD, etc. And any of those can"
  },
  {
    "startMs": 287840,
    "text": "contribute to errors. So we spent time"
  },
  {
    "startMs": 290000,
    "text": "we might be able to reduce that 210 to a"
  },
  {
    "startMs": 292560,
    "text": "smaller number that is I think a known"
  },
  {
    "startMs": 296080,
    "text": "area that we're tackling but I think"
  },
  {
    "startMs": 299040,
    "text": "knob 2 which is what I was talking about"
  },
  {
    "startMs": 301040,
    "text": "is what we'll go through here which is"
  },
  {
    "startMs": 303520,
    "text": "keeping the accuracy exactly the same."
  },
  {
    "startMs": 306080,
    "text": "So 79% what could we do in conditions of"
  },
  {
    "startMs": 310080,
    "text": "uncertainty to improve user satisfaction"
  },
  {
    "startMs": 313280,
    "text": "apparent and I I think we can do a lot."
  },
  {
    "startMs": 315600,
    "text": "So let's start first with the original"
  },
  {
    "startMs": 317919,
    "text": "system is just acting like I said user"
  },
  {
    "startMs": 320000,
    "text": "says something system plays a song it's"
  },
  {
    "startMs": 321919,
    "text": "either the right song or the wrong song"
  },
  {
    "startMs": 324160,
    "text": "immediately I think just common sense"
  },
  {
    "startMs": 326160,
    "text": "tells us that we could introduce at"
  },
  {
    "startMs": 328160,
    "text": "least one system behavior to stop or"
  },
  {
    "startMs": 330479,
    "text": "rather to reject the hypothesis and do"
  },
  {
    "startMs": 333120,
    "text": "nothing. So uh there is now one more"
  },
  {
    "startMs": 336320,
    "text": "option to decide the system may decide"
  },
  {
    "startMs": 338639,
    "text": "and say sorry I didn't get that or sorry"
  },
  {
    "startMs": 341039,
    "text": "could you repeat that? Uh the challenge"
  },
  {
    "startMs": 343360,
    "text": "of course is how how when do we decide"
  },
  {
    "startMs": 347759,
    "text": "to stop and I mean quantitatively. Um"
  },
  {
    "startMs": 351360,
    "text": "here's one approach to kind of"
  },
  {
    "startMs": 353600,
    "text": "visualizing this because if we don't"
  },
  {
    "startMs": 355680,
    "text": "we'll just take probably some swag like"
  },
  {
    "startMs": 357680,
    "text": "some guesstimate and I'll prove that if"
  },
  {
    "startMs": 359840,
    "text": "we just took a guesstimate we would end"
  },
  {
    "startMs": 362160,
    "text": "up with a worse situation than a more"
  },
  {
    "startMs": 364080,
    "text": "rigorous approach. So let's just assume"
  },
  {
    "startMs": 366160,
    "text": "I took those thousand data points and"
  },
  {
    "startMs": 368639,
    "text": "like I said they've been annotated and"
  },
  {
    "startMs": 370479,
    "text": "we assign a confidence score a single"
  },
  {
    "startMs": 373600,
    "text": "confidence score to the hypothesis that"
  },
  {
    "startMs": 376160,
    "text": "was generated by the system you know"
  },
  {
    "startMs": 378160,
    "text": "between zero and one and let's say it's"
  },
  {
    "startMs": 379680,
    "text": "reasonably calibrated. This is a"
  },
  {
    "startMs": 381120,
    "text": "simplification of if it's a cascaded"
  },
  {
    "startMs": 383199,
    "text": "system there are multiple layers and"
  },
  {
    "startMs": 384960,
    "text": "multiple you know confidence scores but"
  },
  {
    "startMs": 386720,
    "text": "let's just assume that for now. Whoops."
  },
  {
    "startMs": 388720,
    "text": "So we're going to have 790 points 200"
  },
  {
    "startMs": 391759,
    "text": "that are correct 210 wrong. Each one has"
  },
  {
    "startMs": 394400,
    "text": "a confidence score and we're going to"
  },
  {
    "startMs": 396080,
    "text": "plot it, you know, plot the"
  },
  {
    "startMs": 397440,
    "text": "distributions. Uh on the x-axis, I've"
  },
  {
    "startMs": 400720,
    "text": "just converted from 0ero to one to"
  },
  {
    "startMs": 402240,
    "text": "percentages. And the question is how do"
  },
  {
    "startMs": 406000,
    "text": "we choose a threshold t such that"
  },
  {
    "startMs": 409440,
    "text": "whatever that percentage is um to the"
  },
  {
    "startMs": 412080,
    "text": "left of it meaning if when the system um"
  },
  {
    "startMs": 415520,
    "text": "forms a hypothesis if the confidence"
  },
  {
    "startMs": 418240,
    "text": "score c is less than that t stop and say"
  },
  {
    "startMs": 421759,
    "text": "sorry otherwise play question is how do"
  },
  {
    "startMs": 424560,
    "text": "we choose a t so far everything I'm"
  },
  {
    "startMs": 426160,
    "text": "saying is fairly common sensical but"
  },
  {
    "startMs": 428560,
    "text": "this is where um intuition will fail us"
  },
  {
    "startMs": 432639,
    "text": "we might say something like okay I don't"
  },
  {
    "startMs": 434639,
    "text": "know let's do 65%. It seems you know gut"
  },
  {
    "startMs": 437759,
    "text": "feeling like okay it's kind of confident"
  },
  {
    "startMs": 439919,
    "text": "that's probably when we should speak. Um"
  },
  {
    "startMs": 442479,
    "text": "now here's where we start coming out"
  },
  {
    "startMs": 444720,
    "text": "with some sophistication."
  },
  {
    "startMs": 446720,
    "text": "Any tea we choose is producing bad"
  },
  {
    "startMs": 449199,
    "text": "outcomes. Bad in the in two fields. One"
  },
  {
    "startMs": 453280,
    "text": "is obviously on the left side anytime"
  },
  {
    "startMs": 455840,
    "text": "you stop it's bad. The user doesn't want"
  },
  {
    "startMs": 458240,
    "text": "it to stop. He wants to they want to"
  },
  {
    "startMs": 460160,
    "text": "hear their song. The other bad is if you"
  },
  {
    "startMs": 463280,
    "text": "do play a wrong song, of course that's"
  },
  {
    "startMs": 466319,
    "text": "bad as well. So these are two two kinds"
  },
  {
    "startMs": 468880,
    "text": "of bad outcomes. But here's the"
  },
  {
    "startMs": 471680,
    "text": "important part. Now I've like elaborated"
  },
  {
    "startMs": 473919,
    "text": "on the um tree diagram on the right hand"
  },
  {
    "startMs": 476960,
    "text": "side. The bad outcomes are not equally"
  },
  {
    "startMs": 480240,
    "text": "bad. They're not the same thing from a"
  },
  {
    "startMs": 482960,
    "text": "user perspective. And obviously let's"
  },
  {
    "startMs": 485120,
    "text": "let's think about it. If the wrong song"
  },
  {
    "startMs": 487840,
    "text": "plays, you said play kiss and it starts"
  },
  {
    "startMs": 491199,
    "text": "playing kiss by Chris Brown instead of"
  },
  {
    "startMs": 494800,
    "text": "the one by Prince. That's going to be um"
  },
  {
    "startMs": 498479,
    "text": "the highest user cost. Now I'm defining"
  },
  {
    "startMs": 501680,
    "text": "user cost from the user's perspective."
  },
  {
    "startMs": 503599,
    "text": "First I have to like hear music and"
  },
  {
    "startMs": 505280,
    "text": "realize that is not Prince. Then I have"
  },
  {
    "startMs": 507919,
    "text": "to shout over my Alexa and um you know"
  },
  {
    "startMs": 511680,
    "text": "get it to stop and then I have to"
  },
  {
    "startMs": 513120,
    "text": "re-request. All of that is a lot of"
  },
  {
    "startMs": 515440,
    "text": "effort. that is definitely a worse"
  },
  {
    "startMs": 517200,
    "text": "outcome than the system stopping and"
  },
  {
    "startMs": 519760,
    "text": "saying sorry I didn't understand that"
  },
  {
    "startMs": 522399,
    "text": "however we should go further and try to"
  },
  {
    "startMs": 525200,
    "text": "quantify that relative badness and there"
  },
  {
    "startMs": 527760,
    "text": "many ways to do it and I think this is"
  },
  {
    "startMs": 529120,
    "text": "an area to be explored for now let's"
  },
  {
    "startMs": 531120,
    "text": "just consider this a heristic of if that"
  },
  {
    "startMs": 535040,
    "text": "outcome happens how many more seconds"
  },
  {
    "startMs": 538000,
    "text": "additional seconds will it take for the"
  },
  {
    "startMs": 539680,
    "text": "user to get back to success which is to"
  },
  {
    "startMs": 541600,
    "text": "play the song they wanted kiss by Prince"
  },
  {
    "startMs": 544240,
    "text": "and I'm I just put down some numbers"
  },
  {
    "startMs": 546480,
    "text": "here. Let's say in the case of a bad"
  },
  {
    "startMs": 548800,
    "text": "song, it's 10 seconds if you add up all"
  },
  {
    "startMs": 550800,
    "text": "the things I got to do. And if it's a I"
  },
  {
    "startMs": 554000,
    "text": "didn't understand you, it's 4 seconds"
  },
  {
    "startMs": 555680,
    "text": "because that's how long it would take"
  },
  {
    "startMs": 556720,
    "text": "you to respe and and the extra latency."
  },
  {
    "startMs": 559519,
    "text": "And now here's where we can start"
  },
  {
    "startMs": 563040,
    "text": "utilizing that. If we go back to our"
  },
  {
    "startMs": 565440,
    "text": "distribution curve on trying to find out"
  },
  {
    "startMs": 567600,
    "text": "where is T. Now we've basically turned"
  },
  {
    "startMs": 571040,
    "text": "this into a problem of minimizing a cost"
  },
  {
    "startMs": 574240,
    "text": "function. It's a user cost function. It"
  },
  {
    "startMs": 576240,
    "text": "is the number of bad acts wherever that"
  },
  {
    "startMs": 578240,
    "text": "whatever the t causes times 10 because"
  },
  {
    "startMs": 580720,
    "text": "that was a unit cost we gave plus the"
  },
  {
    "startMs": 583440,
    "text": "number of stops times four because"
  },
  {
    "startMs": 585040,
    "text": "that's the the unit cost we gave. By the"
  },
  {
    "startMs": 588399,
    "text": "way, one thing I should have elaborated"
  },
  {
    "startMs": 591040,
    "text": "because I work in voice and we like"
  },
  {
    "startMs": 592880,
    "text": "language and we like puns. So this whole"
  },
  {
    "startMs": 595040,
    "text": "thing is called an outcome user cost"
  },
  {
    "startMs": 597920,
    "text": "heruristic. So that spells the word ouch"
  },
  {
    "startMs": 600880,
    "text": "and that is some expression of pain."
  },
  {
    "startMs": 603839,
    "text": "Yes, we are you know language nerds. So"
  },
  {
    "startMs": 606160,
    "text": "these kinds of things amuse us. Um so"
  },
  {
    "startMs": 608640,
    "text": "now let's consider that is the cost"
  },
  {
    "startMs": 610399,
    "text": "function is to minimize the ouch. And"
  },
  {
    "startMs": 612880,
    "text": "now um that let's see if uh I'm going to"
  },
  {
    "startMs": 616720,
    "text": "bring up a tool. Let's see if this"
  },
  {
    "startMs": 618399,
    "text": "works."
  },
  {
    "startMs": 620399,
    "text": "Where I have actually gotten or with one"
  },
  {
    "startMs": 623600,
    "text": "of my coding assistants gotten uh an"
  },
  {
    "startMs": 626720,
    "text": "interactive"
  },
  {
    "startMs": 629040,
    "text": "um graph where we have actually plotted"
  },
  {
    "startMs": 632240,
    "text": "those thousand points and as we vary the"
  },
  {
    "startMs": 636560,
    "text": "threshold t you can see that the total"
  },
  {
    "startMs": 640560,
    "text": "user cost here which is that function of"
  },
  {
    "startMs": 643120,
    "text": "you know x * y + a * b actually changes."
  },
  {
    "startMs": 646480,
    "text": "So let's in the very beginning when we"
  },
  {
    "startMs": 649120,
    "text": "said the system was just playing"
  },
  {
    "startMs": 652160,
    "text": "the the cost across those thousand"
  },
  {
    "startMs": 654640,
    "text": "points was 2100 or divided by a,000 is"
  },
  {
    "startMs": 657600,
    "text": "2.1 ouch points per turn. Then we said"
  },
  {
    "startMs": 661920,
    "text": "okay let's insert a stop behavior and"
  },
  {
    "startMs": 664880,
    "text": "let's like wing it and say 65%. That's"
  },
  {
    "startMs": 667920,
    "text": "when I want the threshold. If we brought"
  },
  {
    "startMs": 670160,
    "text": "this up to 65 yeah that's better. Now"
  },
  {
    "startMs": 673120,
    "text": "it's 1904 or 1.9 per turn, but it's not"
  },
  {
    "startMs": 676800,
    "text": "optimal. As it turns out, if we do"
  },
  {
    "startMs": 679040,
    "text": "actually um ask for the AI to solve the"
  },
  {
    "startMs": 683680,
    "text": "uh the problem across this curve, it"
  },
  {
    "startMs": 685360,
    "text": "turns out 43%. So I'll drag it now to 43"
  },
  {
    "startMs": 690560,
    "text": "is in fact"
  },
  {
    "startMs": 694000,
    "text": "the optimal"
  },
  {
    "startMs": 696640,
    "text": "optimal point of t. This minimizes the"
  },
  {
    "startMs": 699600,
    "text": "cost function. You can see it's the"
  },
  {
    "startMs": 701040,
    "text": "lowest point on this graph down here to"
  },
  {
    "startMs": 703040,
    "text": "1"
  },
  {
    "startMs": 704560,
    "text": "27. So effectively we haven't changed"
  },
  {
    "startMs": 708240,
    "text": "the accuracy at all. The system is not"
  },
  {
    "startMs": 710240,
    "text": "any smarter in that sense. But with some"
  },
  {
    "startMs": 712560,
    "text": "clever system behavior, conversational"
  },
  {
    "startMs": 715279,
    "text": "behavior is what we'd call it and some"
  },
  {
    "startMs": 716959,
    "text": "optimization and a cost function called"
  },
  {
    "startMs": 719839,
    "text": "ouch. Um we have from the user's"
  },
  {
    "startMs": 722800,
    "text": "perspective produced a more satisfactory"
  },
  {
    "startMs": 726160,
    "text": "assistant. And this is not a trivial you"
  },
  {
    "startMs": 728160,
    "text": "know accomplishment. Okay. Now, let me"
  },
  {
    "startMs": 730480,
    "text": "go back to this. [clears throat] Let me"
  },
  {
    "startMs": 732639,
    "text": "see if I can get this. Oh, great. Okay,"
  },
  {
    "startMs": 736720,
    "text": "let's continue this. Let's continue this"
  },
  {
    "startMs": 739839,
    "text": "with by now adding one more behavior."
  },
  {
    "startMs": 742079,
    "text": "Let's call it the confirm behavior. So,"
  },
  {
    "startMs": 743839,
    "text": "there was play obviously, then stop,"
  },
  {
    "startMs": 746000,
    "text": "confirm. Confirm is basically the system"
  },
  {
    "startMs": 748800,
    "text": "after you said something saying uh kiss"
  },
  {
    "startMs": 752399,
    "text": "play kiss by Prince or maybe play kiss"
  },
  {
    "startMs": 755279,
    "text": "by Chris Brown. And uh you know the user"
  },
  {
    "startMs": 758320,
    "text": "can either confirm like affirm it or"
  },
  {
    "startMs": 760480,
    "text": "they can correct it. It is a different"
  },
  {
    "startMs": 762320,
    "text": "kind of behavior and again this is kind"
  },
  {
    "startMs": 764560,
    "text": "of how humans behave. Um that's"
  },
  {
    "startMs": 766399,
    "text": "obviously the inspiration. Now if we go"
  },
  {
    "startMs": 769120,
    "text": "back to our problem of optimization,"
  },
  {
    "startMs": 772480,
    "text": "we have a third obviously um option"
  },
  {
    "startMs": 775760,
    "text": "which is to confirm. And so this would"
  },
  {
    "startMs": 778160,
    "text": "translate to two thresholds"
  },
  {
    "startMs": 780800,
    "text": "um two thresholds which are separating"
  },
  {
    "startMs": 783680,
    "text": "the distribution into three spaces of"
  },
  {
    "startMs": 787920,
    "text": "stop, confirm and uh act. And the"
  },
  {
    "startMs": 792240,
    "text": "question is now where are these T's? and"
  },
  {
    "startMs": 795040,
    "text": "we have now given up on guesstimating"
  },
  {
    "startMs": 796959,
    "text": "because we know it doesn't work. So"
  },
  {
    "startMs": 798480,
    "text": "we're going to be a lot smarter and go"
  },
  {
    "startMs": 801200,
    "text": "back to the concept of user outcome cost"
  },
  {
    "startMs": 805120,
    "text": "and then you know use it go look for"
  },
  {
    "startMs": 807600,
    "text": "some optimization in that graph. So"
  },
  {
    "startMs": 809680,
    "text": "let's uh define what are the what are"
  },
  {
    "startMs": 812320,
    "text": "all the possible bad outcomes that t1"
  },
  {
    "startMs": 814480,
    "text": "and t2 um make for. So good you can see"
  },
  {
    "startMs": 818720,
    "text": "my cursor. So uh of course any stops are"
  },
  {
    "startMs": 822079,
    "text": "still bad. Then in the middle are"
  },
  {
    "startMs": 825440,
    "text": "confirmations. Confirmations are bad"
  },
  {
    "startMs": 827360,
    "text": "because they slow the user down. There"
  },
  {
    "startMs": 829440,
    "text": "is a confirmation outcome called confirm"
  },
  {
    "startMs": 832800,
    "text": "yes where they just affirmed it by"
  },
  {
    "startMs": 834720,
    "text": "saying yeah or no where they had to"
  },
  {
    "startMs": 836959,
    "text": "correct it. And going back to our"
  },
  {
    "startMs": 839600,
    "text": "formula these outcomes are not equally"
  },
  {
    "startMs": 842800,
    "text": "bad. And in fact, nobody will, I think,"
  },
  {
    "startMs": 846480,
    "text": "argue here from a user's perspective."
  },
  {
    "startMs": 848959,
    "text": "Affirming, just saying yes is obviously"
  },
  {
    "startMs": 851680,
    "text": "less painful than saying no and then"
  },
  {
    "startMs": 853760,
    "text": "having to restate whatever it is that"
  },
  {
    "startMs": 855360,
    "text": "you wanted in the first place. So now we"
  },
  {
    "startMs": 857440,
    "text": "I've assigned values of two or six. And"
  },
  {
    "startMs": 859440,
    "text": "again, I said it was a heristic. This"
  },
  {
    "startMs": 861519,
    "text": "would be roughly the amount of time it"
  },
  {
    "startMs": 863199,
    "text": "would take for the extra for the user to"
  },
  {
    "startMs": 865360,
    "text": "get to the song they want. Saying"
  },
  {
    "startMs": 867199,
    "text": "listening and then saying yes is like"
  },
  {
    "startMs": 868800,
    "text": "two seconds. Um and then now"
  },
  {
    "startMs": 874079,
    "text": "uh we restate the cost function for this"
  },
  {
    "startMs": 877760,
    "text": "you know added behavior as this number"
  },
  {
    "startMs": 880959,
    "text": "of you know bad type one times unit cost"
  },
  {
    "startMs": 883760,
    "text": "bad type plus bad type two times unit"
  },
  {
    "startMs": 885839,
    "text": "cost etc. And now we try to minimize"
  },
  {
    "startMs": 889199,
    "text": "this user cost function and minimize the"
  },
  {
    "startMs": 892000,
    "text": "ouch."
  },
  {
    "startMs": 893760,
    "text": "Yes, I'm going to keep doing that pun."
  },
  {
    "startMs": 896160,
    "text": "Um let's go back. So this is now the"
  },
  {
    "startMs": 900240,
    "text": "interactive graph but"
  },
  {
    "startMs": 903440,
    "text": "with"
  },
  {
    "startMs": 906320,
    "text": "um the cost values the unit costs here"
  },
  {
    "startMs": 908720,
    "text": "10264"
  },
  {
    "startMs": 910639,
    "text": "and uh you know we're just going to ask"
  },
  {
    "startMs": 913440,
    "text": "the AI to tell us here's the heat map"
  },
  {
    "startMs": 916240,
    "text": "because it's now two dimensions saying"
  },
  {
    "startMs": 918160,
    "text": "that the optimal values are 41 for the"
  },
  {
    "startMs": 922880,
    "text": "the T1 and 49 for the T2 and if we"
  },
  {
    "startMs": 927600,
    "text": "employed that then we would go to 1464."
  },
  {
    "startMs": 931519,
    "text": "Uh, by the way, whatever numbers I put"
  },
  {
    "startMs": 933440,
    "text": "in here, like let's say I thought wrong"
  },
  {
    "startMs": 936639,
    "text": "act was 20. It's really irritating and"
  },
  {
    "startMs": 939600,
    "text": "painful and takes way longer to actually"
  },
  {
    "startMs": 942079,
    "text": "correct it when you hear a wrong song."
  },
  {
    "startMs": 944160,
    "text": "That would change you know all these"
  },
  {
    "startMs": 945839,
    "text": "numbers uh and the optim optimal point."
  },
  {
    "startMs": 948399,
    "text": "So again it is about how what is the"
  },
  {
    "startMs": 950480,
    "text": "relative badness of these outcomes also"
  },
  {
    "startMs": 952800,
    "text": "of course the distribution curves"
  },
  {
    "startMs": 954639,
    "text": "naturally. Uh let's go back here. Okay."
  },
  {
    "startMs": 958240,
    "text": "So, um I'm gonna"
  },
  {
    "startMs": 962399,
    "text": "speed up a little bit. Uh let's go back"
  },
  {
    "startMs": 966240,
    "text": "here."
  },
  {
    "startMs": 967759,
    "text": "Presentation mode. Okay. So, what have"
  },
  {
    "startMs": 971040,
    "text": "we shown that if we did the super naive"
  },
  {
    "startMs": 973839,
    "text": "approach, it's 2.1 act and stop 1.9 then"
  },
  {
    "startMs": 979360,
    "text": "1.27 then 1.26. We are able to bring"
  },
  {
    "startMs": 982959,
    "text": "this with every added layer of"
  },
  {
    "startMs": 985279,
    "text": "sophistication, adding more behaviors,"
  },
  {
    "startMs": 987680,
    "text": "being smart about outcome, uh, user cost"
  },
  {
    "startMs": 990079,
    "text": "and optimizing. Um, we have made a"
  },
  {
    "startMs": 993600,
    "text": "tremendous difference without changing"
  },
  {
    "startMs": 995279,
    "text": "the accuracy at all. Um, this was a"
  },
  {
    "startMs": 998399,
    "text": "super simplified example. In real"
  },
  {
    "startMs": 1000800,
    "text": "systems, you're not going to have"
  },
  {
    "startMs": 1002480,
    "text": "obviously some offline decision"
  },
  {
    "startMs": 1004399,
    "text": "threshold or two. It's going to be a"
  },
  {
    "startMs": 1006160,
    "text": "real time, you know, learned decision"
  },
  {
    "startMs": 1008079,
    "text": "model. But the principle is the same."
  },
  {
    "startMs": 1010240,
    "text": "And I believe this is uh scalable across"
  },
  {
    "startMs": 1014160,
    "text": "all voice AI surfaces. Obviously this is"
  },
  {
    "startMs": 1016560,
    "text": "a smart speaker but if we go across any"
  },
  {
    "startMs": 1021199,
    "text": "of these surfaces you will find the"
  },
  {
    "startMs": 1022880,
    "text": "equivalence. If we um we will find the"
  },
  {
    "startMs": 1027199,
    "text": "analogies with some differences but the"
  },
  {
    "startMs": 1029520,
    "text": "spirit and the I think the the gain will"
  },
  {
    "startMs": 1033679,
    "text": "be similar. So just for example in the"
  },
  {
    "startMs": 1036559,
    "text": "TV AI assistant space if you employ it"
  },
  {
    "startMs": 1040880,
    "text": "here it's going to you're going to have"
  },
  {
    "startMs": 1042640,
    "text": "the same thing when users express"
  },
  {
    "startMs": 1044400,
    "text": "intents like on TV it's you know open a"
  },
  {
    "startMs": 1047520,
    "text": "channel that's one of the most common"
  },
  {
    "startMs": 1049280,
    "text": "obviously um requests on a TV voice"
  },
  {
    "startMs": 1052000,
    "text": "assistant same thing you're going to"
  },
  {
    "startMs": 1054320,
    "text": "find you'll have exactly the same"
  },
  {
    "startMs": 1055760,
    "text": "approach but the difference will be"
  },
  {
    "startMs": 1058880,
    "text": "maybe in the the assignments of the user"
  },
  {
    "startMs": 1062160,
    "text": "outcomes because the UI and the"
  },
  {
    "startMs": 1063840,
    "text": "modalities are different when you have a"
  },
  {
    "startMs": 1066080,
    "text": "TV you have a multimodal interface where"
  },
  {
    "startMs": 1068880,
    "text": "choices can be shown. So instead of you"
  },
  {
    "startMs": 1071200,
    "text": "know asking did you mean ABC you know uh"
  },
  {
    "startMs": 1075760,
    "text": "news live by speech that you will the"
  },
  {
    "startMs": 1079600,
    "text": "system would display choices and not"
  },
  {
    "startMs": 1081360,
    "text": "just one it show ABC News live this that"
  },
  {
    "startMs": 1083600,
    "text": "would be the confirm step and if it's"
  },
  {
    "startMs": 1085760,
    "text": "visual and you can use your remote"
  },
  {
    "startMs": 1087520,
    "text": "control to select something it's less"
  },
  {
    "startMs": 1090400,
    "text": "pain so you would change some of these"
  },
  {
    "startMs": 1092080,
    "text": "values or if in fact launching the"
  },
  {
    "startMs": 1095200,
    "text": "channel would kick you out of your"
  },
  {
    "startMs": 1096799,
    "text": "current state then it would go in the"
  },
  {
    "startMs": 1098960,
    "text": "other direction than cost of you know a"
  },
  {
    "startMs": 1101200,
    "text": "bad act would go much higher. So it's"
  },
  {
    "startMs": 1104080,
    "text": "the same concept but in this new"
  },
  {
    "startMs": 1106240,
    "text": "modalities"
  },
  {
    "startMs": 1107760,
    "text": "um variables can change, values can"
  },
  {
    "startMs": 1110240,
    "text": "change, arguments can change but the"
  },
  {
    "startMs": 1112400,
    "text": "premise still holds and you can improve"
  },
  {
    "startMs": 1115440,
    "text": "from the user's perspective because"
  },
  {
    "startMs": 1117440,
    "text": "we're all about you know making humans"
  },
  {
    "startMs": 1119600,
    "text": "happy. Um you can make them happier and"
  },
  {
    "startMs": 1124000,
    "text": "this as I said in conclusion can be"
  },
  {
    "startMs": 1126799,
    "text": "applied across all surfaces. I did say"
  },
  {
    "startMs": 1129840,
    "text": "at the very beginning, just to recap for"
  },
  {
    "startMs": 1132000,
    "text": "us, that voice is great when it works,"
  },
  {
    "startMs": 1135760,
    "text": "bad when it doesn't. And as we get into"
  },
  {
    "startMs": 1138960,
    "text": "embodied AI, where these AI assistants"
  },
  {
    "startMs": 1141679,
    "text": "are taking actions, physical or even"
  },
  {
    "startMs": 1144160,
    "text": "digital, like making a phone call or"
  },
  {
    "startMs": 1146400,
    "text": "sending an email, it is getting more and"
  },
  {
    "startMs": 1149840,
    "text": "more difficult just to rely on accuracy"
  },
  {
    "startMs": 1152400,
    "text": "to improve user satisfaction. I believe"
  },
  {
    "startMs": 1155120,
    "text": "there's a whole knob the second knob"
  },
  {
    "startMs": 1157200,
    "text": "called smarter conversational behavior"
  },
  {
    "startMs": 1159760,
    "text": "under uncertainty"
  },
  {
    "startMs": 1161919,
    "text": "and um if we actually exploit that we"
  },
  {
    "startMs": 1165919,
    "text": "can uh very much help these AI systems"
  },
  {
    "startMs": 1170640,
    "text": "reach a acceptable user experience"
  },
  {
    "startMs": 1174240,
    "text": "otherwise I think this will continue to"
  },
  {
    "startMs": 1176799,
    "text": "be a bottleneck like a lot of things"
  },
  {
    "startMs": 1178480,
    "text": "will get better but if the voice"
  },
  {
    "startMs": 1180240,
    "text": "interface as experienced by user does"
  },
  {
    "startMs": 1183280,
    "text": "not improve it is going to be a a a"
  },
  {
    "startMs": 1186320,
    "text": "choke point. And um if you just remember"
  },
  {
    "startMs": 1190640,
    "text": "one word or two words from this whole um"
  },
  {
    "startMs": 1194640,
    "text": "presentation, it would be to minimize"
  },
  {
    "startMs": 1197360,
    "text": "the ouch of the experience. Um so thank"
  },
  {
    "startMs": 1200880,
    "text": "you. I'll stick around for questions if"
  },
  {
    "startMs": 1203440,
    "text": "you guys got any. Thanks a lot."
  },
  {
    "startMs": 1206523,
    "text": "[applause]"
  },
  {
    "startMs": 1221589,
    "text": ">> [music]"
  }
]